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ABSTRACT 

The Texas Automated Student and Adult Learner 
Follow-Up System was developed as part of a larger effort to improve 
and coordinate the delivery of education and training of a skilled 
work force. The primary task of the Follow-Up System in Program Year 
1993-94 was to obtain outcome information on the former students and 
participants of the work force development programs. The process for 
obtaining outcomes data consisted of three phases: seed record 
extraction, record matching, and an employer survey. Results for 
public education were in the form of a prototype report card for six 
pilot districts selected from volunteers. These report cards 
indicated employemnt outcomes and continued pursuit of education. 
Labor market results were fairly consistent across the pilots. A 
pilot study for the Job Training Partnership Act system obtained 
follow-up data through the automated record linkage technique and 
compared results to those obtained through the traditional telephone 
survey techniques by Texas A&M University. The comparison revealed 
nearly identical results. For the third consecutive year, automated 
follow-up documented successful outcomes for 85 percent or more of 
the community and technical college cohorts studied. Recommendations 
for the future were to expand the breadth and depth of coverage and 
degree of automation. (Appendixes include the following: record 
linkage technique; file layouts; sample data sharing agreement; 
guides for using follow-up data; 16 endnotes; and a glossary.) 
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EXECUTIVE SUMMARY 



The Texas Automated Student and Adult Learner Follow-Up System is an integral part of a larger effort 
to improve and coordinate the delivery of an education and training of a skilled workforce. The Texas Workforce 
and Economic Competitiveness Act of 1993 (Senate Bill 642) mandates the use of outcomes information collected 
by the follow-up system in planning and evaluating federally funded programs and services offered by the sute’s 
workforce development partners. Parts of the mandated follow-up system were in place when Senate Bill 642 was 
adopted. The Texas State Occupational Information Coordinating Committee (SOICC), as the designated follow- 
up entity, provided the newly created Texas Council on Workforce and Economic Competitiveness (TCWEC) 
with technical assistance based on the workforce development partners’ collective wisdom as well as information 
about best practices used in other states. In their effort to transition from a collection of disparate 
agency/program follow-up practices to a fully mature, integrated and comprehensive statewide system, the 
SOICC balanced public demands for accountability in education and training with the need for practitioner 
acceptance of the system as fair, reasonable and worthwhile. 

While the SOICC reports directly to the TCWEC, the Automated Student Follow-Up System was 
funded in Program Year 1993-94 by three of the state’s workforce development partners: the Texas Higher 
Education Coordinating Board, the Texas Education Agency and the Texas Department of Commerce. Funds 
were drawn from federal allocations of dollars to the state under the Carl D. Perkins Vocational and Technical 
Training Act and the Job Training Partnership Act. Follow-up activities were governed by a management team 
from those three agencies and operated upon the advice of practitioners. Five site based committees were formed 
across the state to solicit practitioner advice. Motions and resolutions were carried forward from the site-based 
committees to a Steering Committee whose representation was balanced geographically and by constituency. 

The SOICC coordinated record’ matching activities to obtain outcome information and managed an 
employer survey subcontract to obtain more detailed occupational employment information. (The employer 
survey represented a nearly tenfold increase in the number of records processed compared to the prior year’s 
subcontract.) The SOICC provided baseline analysis of follow-up data and facilitated distribution of data files to 
service providers. The SOICC rendered technical assistance to the TCWEC, tri-agency personnel, and local 
service providers to foster better understanding and more widespread use of follow-up information in a process 
of continuous program improvement. 

In addition, the SOICC collaborated with agency personnel and practitioners to improve analytic tools 
and information report/display formats that make the data more meaningful and useful. The SOICC strove for 
consensus in resolving technical differences among tri-agency evaluation practices and procedures by engaging 
practitioners in dialogues at association meetings, workshops and a statewide conference. Upon the advice of 
practitioners, the SOICC negotiated additional data sharing agreements to increase the system’s capacity to 
document a wider range of successful outcomes. 

As a result of these efforts, the follow-up system accomplished two primary objectives. Services to 
existing customers (the tri-agencies) were improved. As current service improved, the follow-up system became 
more attractive to the state’s other partners in workforce development: the Texas Employment Commission and 
the Texas Department of Human Services. The achievements of the Texas follow-up system were brought to the 
attention of administrators and practitioners in other states and were influential in shaping common approaches 
to shared problems. Thus SOICC activities under the 1993-94 grant moved the follow-up system closer to the 
model envisioned for the state in Senate Bill 642 while also providing coordination with out-of-state activities to 
ensure a greater degree of cross-state comparability. 
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OVERVIEW 



Automated student and adult learner follow-up is designed to answer a fundamental 
question: "What happens to students and participants after they receive publicly fund^ education 
and training?". The question is important to divergent interested parties for a variety of 
overlapping reasons. 

• Taxpayers want to know and deserve to get a reasonable return on their investment 
in education and training programs. This perspective is refleaed in the federal 
Government Performance and Results Act of 1993 (hereinafter, GRP A). 

• Public officials, employers and economists perceive the connection between availability 
of a competent workforce and a community or region’s ability to compete in a global 
economy. This perspective is reflected in theoretic and academic literature such as 
America’s Choice: high skills/low wages ^ and Ray Marshall and Marc Tucker’s Thinking 
for a Living .^ 

• To respond to public demands for accountability in education and training, legislators 
and administrative agency personnel need documentation of successful outcomes to 
evaluate compliance and policy impaa. This perspeaive is refleaed in the federal call 
to outcomes reporting requirements in the Carl D. Perkins Vocational and Technical 
Education Aa (Perkins) and the Job Tr^ning Partnership Aa 0TPA) and in this 
itate’s Workforce and Economic Compaitiveness Aa (SB 642). 

• If planners are to evaluate and constantly improve programs, they need reliable and 
valid outcome data rather than markaing/ recruiting hyperbole, speculation or assump- 
tions about what "ought to work in theory" or non-representative anecdotal informa- 
tion. This sentiment is refleaed in: 

the state’s substantial investment in development, distribution, technical support 
for and reliance on SOCRATES, the automated planning model from the Texas 
State Occupational Information Coordinating Committee (SOICC). 

administrative rules and guidelines such as those issued by the Texas 
Department of Commerce (TDoC) to JTPA Substate/Service Delivery Areas 
(SSAs/SDAs) for Title II and Title III programs; 

institutional self-study guidelines for accreditation reviews by the Southern 
Association of Colleges and Schools (SACS); 

institutional effeaiveness guidelines issued by the Texas Higher Education 
Coordinating Board (Cooidinating Board) and the "VOC116R" and "CB116" 
reporting requirements for public education and higher education established 
respeaively by the Texas Education Agency (TEA) and the Coordinating Board. 



• If training providers and professional educators are to keep the curriculum engaging 
and responsive to labor market demands, they need a constant source of detailed feed- 
back about how well they served their students in general and how effective they were 
in tailoring delivery to the needs of certain special populations. This sentiment is 
reflected in the creation of "Institutional Effectiveness", "Institutional Research" and 
"Quality Assurance" offices within secondary and postsecondary educational 
institutions. It also is evident in the widespread attention given to these topics at 
association meetings, conferences and workshops organized by education and training 
professionals. 

• As students and training program participants stand on the brink of making important 
career decisions, they need hard evidence that selection of a coherent sequence of 
courses will provide a viable pathway to promised or expected successful outcomes. 
Such successes are best predicted by the education and training providers’ track records 
and the results achieved by those who preceded the current cohort along each 
alternative pathway. These sentiments are reflected in the Student Right to Know Act 
and Campus Security Act of 1990 (Public Law 101-542, hereinafter "Right to Know 
legislation”) and in the state’s investment in the SOICC’s development, distribution and 
technical support for Texas CARES (an automated career information delivery system). 
Improved Career Decision-Making seminars, the Career Information Hotline, and 
publications. 

Regardless of focus, all interested parties need answers to the same central question. 
Getting answers would be easy if education and training programs were small and their 
graduates displayed little geographic mobility or employment-skills portability. A telephone 
call or a mailed survey to a former student/participant’s last known point of contact would 
likely reach the appropriate person and elicit a response. 

However, program enrollments have mushroomed, average job tenure with a single 
employer has decreased, and the workforce has become more mobile geographically at the 
same time as the demand for outcomes data from all quarters has increased and analyses have 
extended to include ever increasing timeframes in dehnitions of "long term" success. Moreover, 
as the proliferation of providers intensities competition for scarce education and training 
dollars, programs are increasingly likely to be judged under the public microscope by not only 
the measures they devised and applied to themselves but also by comparisons to other 
programs with similar but not identical missions and subject to somewhat different measures. 
The demand for hard evidence of accountability and comparative performance is often over - 
whelming - beyond the resources available to education and training providers for data 
collection and the requisite analysis to turn raw data into useful information. 

Texas has devised an automated student and adult learner follow-up system to meet the 
information demands and needs of interested parties. 
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By using cost-effective record matching techniques to link to administrative data bases 
constructed and updated constantly for other purposes, it capitalizes on other mandated 
public expenditures for data collection and program administration. 

By integrating services through a central administrative entity, the follow-up system 
eliminates duplication of effort among education and training-providers while reducing 
the burden on data providers. 

By housing integrated services in a neutral, independent entity, the follow-up system 
assures greater objectivity in data collection while building consensus for the fair 
comparisons of performance data on a level playing field through such activities as: 

establishing common performance-data definitions applicable across the state’s 
workforce development partners; 

collecting data through a standard methodology in a shared timeframe; 

creating standardized report formats that make cross-program comparisons easier 
and more meaningful; and 

developing evaluation tools which strike a balance between the expertise and 
experience ^ut unavoidably self-interested orientation of education and training 
providers) and the hard-nosed "bottom-line" market mentality appropriate to 
those who hold them accountable. 

By providing continuity of administration, the follow-up system builds historical files 
to better determine the long term trends which impact program delivery and effective- 
ness and from which realistic yet ambitious performance standards can be derived. 

By bringing together professional staff, subcontractor(s) and advisory committees of 
practitioners and data users, the follow-up system provides the technical assistance 
necessary to translate raw data into user-friendly information tailored separately for the 
needs of administrators/planners, training providers, career guidance counselors/case 
managers, parents, students/ participants, economic development specialists, employers, 
and the general public. 
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BACKGROUND 



1) Automated Studmt and Aduk Learner Folkw-Up as a Key Ekment in a Collective Effort to 
Improve and Coordinate the StaU’s Worlrforce Development Programs 

By revising the State Master Plan for Career and Technical Education (Master Plan) in 
1993, the Coordinating Board, TEA and TDoC committed to improving programs for educa- 
ting and training the workforce, coordinating efforts, and for holding themselves accountable 
• for a reasonable return on the investment of public dollars therein. Revision of the Master 
Plan was not the first cooperative undertaking of the tri-agencies. Beginning with three funded 
pilot projects in 1987-88, the tri-agencies supported creation of regional Quality Work Force 
Planning (QWFP) Committees. These committees bring together public and private sector 
representatives to plan the integrated delivery of career and technical education and training 
and to eliminate unnecessary duplication of programs. They work to ensure that adequate 
resources will be shifted to programs crucial to a region’s prosperity as demands for 
occupational employment change and as new high skill/high wage occupations emerge. They 
also promote dialogue between employers and training providers to ensure that the career and 
technical education .curriculum will produce skilled workers who can demonstrate compe- 
tencies to world class standards. 

Cooperation and coordination is driven both by a common desire to improve the 
state’s economic competitiveness through workforce development and by demands for accoun- 
tability in education. The Workforce and Economic Competitiveness Aa (Senate Bill 642, May 
1993) is the most recent expression of those dual concerns. With passage of SB 642, Texas is 
moving rapidly toward implementation of a statewide, comprehensive automated student and 
adult learner follow-up system. 

Prior to passage of that Act, the tri-agencies (both jointly and individually) funded the 
regional QWFP Committees and enhanced their effectiveness through various capacity 
building efforts. An automated regional planning model, SOCRATES, was developed, 
continuously improved, installed and supported at host-sites to guide QWFP Committees and 
each region’s education and training providers in targeting critical demand occupations. The 
tri-agencies supported a clearinghouse that facilitates exchange of critical information, ideas and 
innovations. The tri-agencies also promoted the concept of Total Quality Management and 
a model for its implementation in the education and training system: site-based management. 
More recently, the tri-agencies agreed to support joint development of: 

• an automated career information delivery system; 

• a process for skill standards identification; and 

• model/demonstration projects for one-stop delivery of education and training services. 




Page 4 



Texas State Occupational Information Coordinating Committee 

12 



Tri-agency implementation of the statewide Tech-Prep High School and Associate 
Degree initiative is noteworthy evidence of that cooperation and commitment across all levels. 
Development of Tech-Prep programs was funded largely with federal dollars under the Perkins 
Act. State level coordination was necessary in setting core standards and measures, in removing 
barriers to streamlined articulation and in cataloging equivalencies across various coherent 
sequences of courses. Identifying significant regional occupational employment demands, 
negotiating specific articulation agreements, curriculum development, and securing employer 
buy-in to specific Tech-Prep degree programs are the responsibility of regional consortia. 

Interagency agreements, coordinated planning, creation of regional forums, mutual 
endorsement of or willingness to test innovative ideas and formal approval of degree programs 
and training service contracts, however, are no guarantee of successfyl outcomes. These must 
be coupled with: 

• a feedback mechanism that constantly monitors outcomes and translates data into 
meaningful information to drive program improvements; 

• a process for revising, deaaivating, or sunsetting programs that do not meet core 
standards; 

• a means of identifying and promoting broader adoption/adaptation of exemplary 
programs, effeaive education and training materials, and successful delivery methods; 

• a mechanism for constantly monitoring externalities - such as changes in the labor 
market, emerging technologies and economic conditions to anticipate concomitant 
employer expeaations; 

• a strategy to help students and adult learners form reasonable career expectations and 
access the most appropriate and effective education and training alternatives; and 

• an accountability process that documents successful outcomes and demonstrates to 
policy-makers and taxpayers that the return on investment is reasonable. 

A comprehensive statewide automated student and adult learner follow-up system is the 
ideal feedback mechanism to keep workforce development partners focused on continuous 
improvement. Such a system must be built carefully. The data colleaed must be valid and 
reliable. Secondly, the analysis and report formats must distill follow-up data into information 
that can be understood readily and unequivocally and translated into recommendations for 
program improvement. Lastly, the follow-up system must be cost-effective. 

Significant program improvements can take place only at the local education and 
training delivery level. Unless follow-up information is available to and embraced by the 
education and training providers, it will have no impact on programs, will produce few 
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benefits for students and adult learners and will make no significant contribution to economic 
competitiveness. If automated follow-up is feared as a punitive, externally imposed system or 
perceived essentially as a compliance reporting requirement, it will be given minimal attention 
at best - if not covertly resisted. In the long term, the cost-effectiveness of the follow-up 
system depends as much - if not more - on its wide spread acceptance as it does on the elegance 
and precision of its methodology. 

The tri-agency funded Automated Student and Adult Learner Follow-up grant for 
Program Year 1993-94 was designed to cultivate widespread acceptance among all constituent 
elements while perfecting a standard methodology that is valid and reliable, can be applied 
consistently across the constituencies and which yields analyses that are useful. The statement 
of work, advisory structure and budget were designed to build a cost-effective follow-up system 
that will have a genuine impact on the economic competitiveness of Texas and the well-being 
of students and adult learners. 

2) The Model: Budding on the Experience of Automated Follow-Up for Higher Education in Texas 

The follow-up system is built upon the collective wisdom of practitioners, 
administrators and tri-agency personnel. It measures factors that professional educators and 
trainers have defined for themselves as important indicators of successful performance. Core 
features of the automated follow-up system were developed over several years by the 
Coordinating Board in cooperation with progressive community and technical colleges and 
innovative individuals. It is effective in producing real change in the curriculum and delivery 
of education and training because community and technical colleges understand that the 
methodology is sound and because the system is perceived not as an externally imposed 
compliance requirement but as a system in which they provided substantial leadership in 
design for use in institutional effectiveness and program evaluation. 

North Harris/Montgomery Community College (NHMCC), using Perkins funds, 
piloted the first automated record matching process for student follow-up in Texas in 1988-89. 
NHMCC constructed its own seed file for matching against the UI wage-records. (There was 
no provision in the NHMCC pilot for automating the identification of former students who 
continued their training at another institution of higher education.) Results from the match 
were used in NHMCC’s self-study report for SACS accreditation review. 

In 1989-90, Andersen Consulting, under a contract to the SOICC, reviewed available 
Texas data bases and procedures developed in other states - primarily Florida - for identifying 
labor market outcomes. Andersen Consulting’s feasibility study recommended a three-phase 
plan for implementing an integrated statewide automated student follow-up system serving 
multiple constituencies through a central administrative entity. 

On a voluntary basis, Bra2osport College joined with NHMCC and 12 others commun- 
ity colleges during the 1990-91 program year to further expand and enhance the piloting of 
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automated UI wage-record matching. Each college created its own seed files. The Coordinating 
Board eliminated error records, combined the files into a master tape and facilitated matching 
of those files against TEC’s UI ws^e-records. Each volunteer college paid its pro rata costs for 
that matching. The Brazosport volunteer pilots recommended that seed files be extraaed from 
the Coordinating Board’s master enrollment files and that the higher education data base be 
included in the match as a means of identifying former students who transfer to continue their 
education and training. 

Community and technical college interest in automated follow-up increased as the 
Brazosport Study volunteer colleges disseminated information about the process and explained 
the value of follow-up data. The University of Texas at Austin served under contraa to the 
Coordinating Board to administer expansion of follow-up services to all community and 
technical colleges. The 1991-92 study matched seed files for all community/technical college 
graduates and non-returners gainst both the UI wage-records and the Coordinating Board’s 
master enrollment files. Workshops and a statewide conference were conduaed during the 
1991-92 program year to ensure that administrators and institutional researchers understood 
both labor market outcome data and information about former students continuing their 
higher education elsewhere. These workshops and the state conference stressed the use of 
follow-up data in evaluating institutional effeaiveness and shaping program improvements. 

Rapid expansion of services to the community and technical colleges occurred because 
administrators and institutional researchers from volunteer colleges took the lead in informing 
their peers about the value of follow-up data. Because they realized, in the long run, that an 
automated record matching process would be the most cost-effeaive means of colleaing data, 
participating administrators and researchers encours^ed the Coordinating Board to allocate 
capacity building funds for a more comprehensive follow-up system. With the 1992-93 contraa 
awarded to the Texas SOICC, these same individuals aaively recruited volunteers to 
participate in piloting other follow-up aaivities: an employer follow-up survey, an adult 
vocational pilot, a public university pilot, a study of former high school 2+2 students, and 
voluntary participation by private universities. 

Understanding and acceptance of automated follow-up has been promoted not only 
through Coordinating Board and SOICC sponsored workshops, conferences and regular 
channels of communication but also through peer-to-peer exchanges: for example, at the Texas 
Association of Institutional Researchers’ annual conference (TAIR), at the Texas Association 
of Post-Secondary Occupational Education Administrators’ semi-annual meetings fTAPSOEA) 
and through informal discussions among community and technical college representatives at 
Quality Work Force Planning committee funaions (QWFP), Tech-Prep consortia meetings, 
or Perkins Committee of Praaitioners’ meaings. 

The concept of follow-up is familiar to some administrators, planners and researchers 
in universities, public education and the Job Training Partnership Aa (JTPA) system. 
However, understanding, commitment to and effeaive utilization of follow-up data do not 
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seem as widespread in these constituencies as they are among community and technical 
colleges. 

While university administrators appreciate the need to understand transfer patterns with 
community and technical colleges and/or other senior institutions, the value of labor market 
outcomes data is less clear to them - especially for their programs that are not occupationally 
specific. The follow-up system developed for community and technical colleges, therefore, 
must be tailored to accommodate the needs and interests of universities. 

Local education agencies (LEAs) must report follow-up information on former 
vocational education students (VOC116 reports). Their reports are based on time/resource- 
consuming telephone and/or mail surveys which may be seriously flawed. Such surveys often 
suffer from low response rates, dependence on unverifiable self-reported behaviors and refusal 
of former students to give their consent to employment verification. Because they are of 
questionable validity and reliability, such follow-up data are seldom subjected to in-depth 
analysis. They carry little weight in the evaluation of and planning improvements for public 
education programs. A more effective feedback mechanism for improving public education 
programs can be developed as automated record matching reduces the costs of acquiring more 
reliable and valid information and as outcome information is explained in terms of other data 
elements already in the Public Education Information Management System (REIMS). 

The Texas Job Training Partnership Act 0TPA) system is required by the federal 
Department of Labor to conduct follow-up of former participants. The data collection costs 
exceed $330,00 per year and are funded with Service Delivery Area (SDA) administration 
dollars. Since there is a ceiling of 20% on the use of JTPA dollars at the SDA level for 
administration and since there are other functions competing for those scarce administration 
dollars, few funds are left for analysis of follow-up data at the SDA level in sufficient detail 
to guide program improvements. Use of follow-up data in the Texas JTPA system, therefore, 
is confined largely to compliance reporting and year-end allocations of incentive funds. As in 
public education, follow-up information will have more impact in the JTPA system if the costs 
of data acquisition are reduced and if SDA personnel learn to couple follow-up information 
with data elements in their Management Information Systems (MIS) to evaluate performance. 

A comprehensive, statewide follow-up system (to include all the above constituencies) 
is envisioned in the Workforce and Economic Competitiveness Act (§2.09). It requires collec- 
tion of sufficient data to draw inferences about the effectiveness of specific programs. Funds 
available under terms of the tri-agencies’ 1993-94 agreement ($225,000) were not sufficient to 
achieve that level of detail for all three constituencies. Moreover, given the prior experiences 
with and involvement in various follow-up activities, the three agencies were not at the same 
level of technical development and constituent acceptance. Services and activities under the 
1993-94 tri-agency contract were, thenfore, desigyied to balance the disparate experiences and needs 
of the three agencies and their constituents with their common interest in moving rapidly into a 
fully integrated and comprehensive statewide follow-up system. 
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Senate Bill 642, §2.09 implies that the SOICC build upon and enhance existing 
components to institutionalize a comprehensive and integrated follow-up system to serve all 
the state’s workforce development partners under the supervision of the Texas Council on 
Workforce and Economic Competitiveness (TCWEC). Parts of the system envisioned in Senate 
Bill 642 were already in place. Transitioning to full, statewide implementation necessitates 
making existing components more productive while building capacity for constituencies not 
yet served. The 1993-94 follow-up grant was designed to help estimate the costs of and address 
issues associated with full-scale, high productivity implementation in subsequent years while 
building confidence among program administrators, educators and training providers, 
prospective students and participants, taxpayers, and employers in the produas and services 
of the follow-up system. The Coordinating Board, TEA, and the TDoC each contributed 
$75,000 toward the transition costs for Program Year 1993-94. 

In addition to targeted services, each constituency benefitted indirectly from follow-up 
services to the other two. For example, a substantial portion of Texas high school graduates 
pursue additional training at the state’s public institutions of higher education and the JTPA 
system purchases off-the-shelf training from community and technical colleges for Title IIA 
and Title III participants. Follow-up infoimation on former higher education students helps 
JTPA case managers and high school counselors provide better career guidance and referrals 
for eligible participants and students. As all three constituencies work toward a seamless 
education and training delivery system - as exemplified by Tech-Prep programs - sharing 
follow-up data will stimulate development of or improvements in agreements where desired 
levels of articulation have not been achieved. Therefore, steps were taken under the 1993-94 
grant to transform the system initiated by community and technical colleges into a joint 
venture among three of the state’s workforce development partners. 

All three constituencies benefited from working through SOICC as a neutral, central 
administrative entity for automated follow-up. With SOICC’s technical assistance, the JTPA 
system and public education avoided pitfalls encountered earlier by trial and error in higher 
education. Standardized data definitions and follow-up methodology - established through 
grassroots consensus building - made it easier to communicate and compare data across 
programs and agencies. The consensus-building approach engendered trust that follow-up data 
will be used in a consistent and even-handed fashion to improve programs across all three 
constituencies. In sharing resources to design a single follow-up system, each constituency 
avoided unnecessary duplication of effort. Effective programs were identified and information 
about them was disseminated to encour^e emulation and adaptation. 

By working through local committees, the SOICC enhanced the principles of site-based 
implementation of Total Quality Man^ement. Confidence in and reliance upon the follow-up 
system is built from the bottom up through peer-to-peer (rather than authoritarian, top^Iown) 
communications. 
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In addition to building upon the prior work of the tri-agencies and the recom- 
mendations of practitioners, the Texas SOICC established communications over the years with 
the other state OICCs, the National OICC, the academic research community, and other 
states* follow-up entities. Through these contacts, common problems and pitfalls were 
identified; information about best practices also was shared. The SOICC, using its Follow-Up 
Steering Committee as a sounding board, forwarded out-of-state experts* recommendations for 
consideration in Texas. The SOICC also participated in joint planning of strategies in a 
concerted effort to secure federal action where necessary and to broaden consensus on methods 
and definitions that would result in the greatest possible degree of cross-state comparability of 
findings. 

The intent in all these activities was not to concoct a follow-up system independently 
and externally impose it upon practitioners. Rather, the intent was to build upon the 
historical achievements and collective wisdom of the practitioners, resolve technical differences 
among the partners* practices, and blend in the experiences based on trial and error of other 
states* efforts. The intent was twofold: 

• to move from demonstration status to a fully matured system capable of delivering 
comprehensive and integrated services to its current customer base; and 

• to bring other workforce development partners (identified in Senate Bill 642) into the 
system. 

3) Evaluating the Performance of the Follow-Up System 

A vision of a fully mature system was developed based on practitioner opinions, both 
from within the state and from the SOICC*s out-of-state advisors. A mature system was 
conceptualized as multi-dimensional. For each dimension, ideal performance levels were 
identified and compared to current practices or achievements to date. Strategies were then 
developed to move from current performance levels to the ideal on each dimension. Resources 
were too scarce to perfect the system in a single program year; therefore, activities under the 
1993-94 grant were scheduled to make the most effective use of available time and resources. 

The SOICC measures and assesses its performance in terms of progress made on each 
dimension. The chart on the next page lists the desired traits of a mature, comprehensive and 
integrated system on the right side. (Each dimension is explained in more detail in the 
following pages.) Practices as of the start of Program Year 1993-94 are located on each 
continuum. This chart serves as a point of departure for self-evaluation of the SOICC*s 
performance under the 1993-94 grant, its concluding remarks about the follow-up system*s 
achievements during the program year, and recommendations for future action. 
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Breadth of Coverage 



System development usually begins with demonstration projects at volunteer pilot sites. 
A transitional system uses representative samples and statistical techniques to draw inferences 
about relevant populations. A more mature system provides fpllow-up on the entire relevant 
population ("universe"). A fully mature system approximates the classic experimental design 
by including studies of comparison groups that did not receive the services provided to parti- 
cipants. 

Automated student follow-up started in Texas with a single volunteer community 
college district then grew to include 13 community and technical colleges. In the third year 
the University of Texas at Austin facilitated record matching for all conununity and technical 
colleges. In 1992-93, the SOICC continued the work begun by UT - Austin on behalf of the 
community and technical colleges. East Texas Sate University joined as the pilot for four year 
institutions of higher education. Leander High School joined as the public education (2+2) 
pilot. Four conununity and technical colleges submitted seed records on their adult vocational 
students. Breadth of coverage, therefore, is uneven among the tri-agencies with the 
Coordinating Board having more well developed practices. 

Degree of Automation 

Follow-up studies are traditionally conducted through telephone and mail surveys using 
contact and background information kept in manual filing systems. These tend to be costly 
and elicit low response rates. With low response rates come response-set biases. The error 
factor in follow-up of this sort is compounded when participants are asked to self-report 
outcomes. Non-automated follow-up requires manual data entry. Here again, the potential for 
error is increased. A fully mature system reduces effort and eliminates some potential for error 
on both ends by automating the generation (or extraction) of seed records and the docu- 
mentation of successful outcomes. 

Public entities collect a wealth of information for a variety of purposes. Some of the 
data are relevant to questions educators and training providers ask. Tapping into those data 
bases helps eliminate unnecessary duplication of effort by capitalizing on other entities’ sunk 
costs in data collection. One expects the quality of the data and the coverage to be better than 
that obtained through traditional methods. Automation of the process eliminates several 
manual data entry steps. It also permits "edit check" programming to capture some errors. 

The follow-up system currently mixes automated record matching and an employer 
survey. It matched seed records for former students against the Coordinating Board’s master 
higher education enrollment files and the Texas Employment Commission’s Unemployment 
Insurance wage-records. In addition, Jim Reed of the Texas Student Information System 
(TEXSIS) conducted employer surveys to obtain more specific data. (The employer survey is 
necessary because the UI wage-records do not contain occupational titles, hourly wages or 
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work-site location data.) The SOICC also conducted negotiations to improve the breadth of 
coverage of record matching by gaining access to federal civil service, postal service and 
military enlistment data. Remaining gaps must be filled by gaining access to other admin- 
istrative data bases. In increasing breadth of coverage, the degree of automation also increases. 

As displayed atop the continuum, technical advances enhance automation by increasing 
the storage capacity, retrieval speed, cross-platform communications, and computational 
capabilities of computers while decreasing the run-time costs and providing an operating 
platform for developing more sophisticated and user-friendly tools. The system, however, is 
not fully integrated. The data bases tapped by the SOICC are not networked together. While 
seed record extraction, appending of outcomes data, and data analysis are done on high speed 
computers, physical transportation of data files is necessary at various stages in the process to 
facilitate record linkages. 

Depth of Coverage 

Automated follow-up originated as a one-time response to a SACS accreditation visit 
to North Harris/Montgomery Community College. Snapshots had to be retaken periodically 
thereafter because changing external factors significantly impact outcomes. As the knowledge 
base grew, researchers refined their questions to better understand the determinants of success. 
Over time, data from a number of snapshots increased. As the picture became more complete, 
more sophisticated questions arose. In particular, researchers want to identify the delayed 
effects and long term results of education and training. The ideal method for collecting follow- 
up data is a longitudinal design that tracks each cohort through multiple points in time. 
Longitudinal studies are better suited to answering more sophisticated and pertinent questions: 
e.g., for determining pre/post participation gains and for identifying the lagged (delayed) 
benefits of program participation. Moreover, other research indicates that our linear models 
of education, training and career path advancement may be inappropriate. Only with 
longitudinal data can we build a model with improved predictive ability. 

Senate Bill 642 mandates longitudinal follow-up at the one, three and five year post- 
program intervals. To date, however, the SOICC is funded only to conduct studies at a single 
point in time for each cohort being tracked. To achieve maximum depth of coverage, the 
system will require more funds. 

Tool Development 

Frequency distributions answer the basic question, "What happened to former students 
and participants after they exit an education and training program?". Once researchers and 
administrators know what happened, they begin to ask: "Why?" "What worked?" "Did the 
program work better for some subgroups than for others?" "Under what conditions?" These 
questions require additional calculations and analyses: percentages, cross-tabulations, regression 
and analysis of variance techniques, for example. If something more sophisticated than a 



Final Reporv Automated Student and Adult Learner Follow-Up, 1993-94 



Page 13 



frequency distribution is needed, consensus on the appropriate calculation or tool(s) to use may 
evaporate. Something as seemngly simple as computing a percent^e may lead to disputes over 
what belongs in the denominator and numerator. Service providers, resizing their programs 
may be effeaed by the way data are displayed and performance is measured, instinctively 
prefer calculations and techniques which put them in the best light. Consensus is restored 
through appeals (to professional objeaivity and the bona fide desire of all to serve their 
customers well) based on evidence that refinements in analytic tools improved their reliability 
and validity. Over time, consistency in results and perceived fairness of the process build 
confidence in the tools. The tools themselves are validated ^y investigating systematic sources 
of error and convergent validity techniques). Statistical adjustments can be developed to correa 
for known sources of error. Conscientiousness, objeaive empirical testing and professionalism 
contribute to the confidence others have in the tools constructed and, in turn, lead to the last 
(but most important) element in a mature system. That is, the data are transformed into 
information that can and will be used. 

Utilization 

In a fledgling system, only a limited number of people are aware of follow-up data. 
Specially trained researchers may know the data intimately but key administrators want it in 
an executive briefing/summary format, use it quickly for limited purposes, then move on to 
other work. Thus, performance data have tremendous untapped potential to effectively guide 
decision-making at a number of other levels: program planning, recruitment, career counseling, 
job placement, etc. Those who could benefit from access to and familiarity with follow-up da ta 
are either unaware of its existence or, if aware of it, have difficulty obtaining and deciphering 
it. As follow-up information is more widely distributed, more praaitioners will appreciate its 
potential uses. Extensive technical assistance improves their understanding of the data and 
increases their confidence in them. With increased confidence comes the inclination to use the 
data to drive program planning and evaluation; with technical assistance comes the under- 
standing necessary for their proper use. 

In a fully mature system, pertinent information is distilled from detailed and 
sophisticated analysis. (The Coordinating Board, for example, uses follow-up data in its in its 
institutional effectiveness model and program evaluations.) The information is translated from 
technical language and jatgon into easier to read yet meaningful summaries for distribution to 
the general public. It is used to explain how decisions were made and supported with hard 
evidence. Succinct report formats and graphics in hardcopy and elearohic formats engage the 
"attentive publics" (i.e., concerned and interested citizens) who use the information to form 
judgments about the return they get for the investment of their tax dollars in education and 
training. As follow-up reports are more widely distributed and consumed, the environment 
in which education and training decisions are made becomes more rational. "Data dialogue" 
replaces expressions of vested interest, guesswork, reliance on selectively-reported or unrepre- 
sentative anecdotes, and/or marketing hyperbole in the decision-making process. 
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1993-94 PROGRAM YEAR GRANT ACTIVITIES 



1) Funding 

The SOICC was awarded a grant of $225,000 to operate the Automated Student and 
Adult Follow-Up System. TEA, TDoC, and the Coordinating Board each contributed $75,000 
for system development and administration. Both TEA and the Coordinating Board 
contributed federal Perkins dollars; TDoC contributed JTPA Title IIA and Title HI dollars. 
The Statement of Work and a line item budget were established in collaboration with the 
SOICC. Additional program income was raised through registration fees for three regional 
workshops and a statewide conference. Earned program income was spent on follow-up activi- 
ties. Expenditures were governed by the Statement of Work, the budget, tri-agency and federal 
fiscal rules and regulations. 

An employer survey subcontract was issued through the competitive bid process to Dr. 
Jim F. Reed (doing business as the Texas Student Information System or TEXSIS) in the 
amount of $101,927. The subcontract amount was later amended with the addition of $5,580 
to ensure that the persistence level for the 1993-94 subcontract matched that provided in 
Program Year 1992-93. Copies of the budget, subcontract budget and subcontract amendment 
are provided in Appendix VIII. 

There is a three year limitation on the use of Perkins dollars for state demonstration 
and capacity building activities. On July 1, 1994, the Texas SOICC will enter its last year as 
a recipient of Perkins dollars for conducting automated follow-up. Part of the SOICC’s 
responsibilities, in anticipation of ineligibility for Perkins dollars after that point, was to 
develop a legislative plan to secure permanent funding for the follow-up system for Program 
Year 1995-96 and beyond. A synopsis of the legislative plan appears in the "Recommendations" 
section of this report. The chief aims of the draft legislative plan are to improve the depth 
and breadth of cover^e of the follow-up system by including participation of additional work- 
force development partners and by securing sufficient funding for mandated longitudinal 
research activities on behalf of all partners. 

2) Governance 

The statement of work, major duties and tasks, and master calendar were developed in 
collaboration with a tri-agency management team: Dr. Mark Butler CI’RA), Robin Campbell 
(TDoC), and Helen Giraitis (Coordinating BoarcQ. The tri-agencies jointly funded other 
workforce development activities with each agency taking lead responsibility for fiscal 
management of one or more of the related grants. The Coordinating Board was responsible 
for managing the Automated Student and Adult Learner Follow-Up System grant for Program 
Year 1993-94. Operations were managed by a system director employed by the SOICC. 
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To ensure responsiveness to customer needs, a committee structure was designed to 
maximize input from practitioners. Five site-based committees were established: Amarillo, 
Beaumont, Dallas, Houston and McAllen. (Membership lists are included in Appendix VII.) 
The System Director convened the first meeting of each site-based conunittee to explain 
responsibilities and the channels of communication to be used for forwarding ideas and 
suggestions for system development and activities. Each site-based committee elected a chair 
and vice-chair. Responsibility for convening subsequent site-based committee meetings was 
turned over to chairpersons. The System Director and the employer survey sub-contractor 
served thereafter as resource persons at the disposal of the site-based committees. The chair and 
vice chairperson of each site-based committee became members of the statewide Steering 
Committee. Additional persons were added to the Steering Committee to balance constituency 
representation, to provide continuity, and to take advantage of the special expertise and 
experience of progressive individuals who were instrumental in the evolution of the follow-up 
system prior to pass^e of Senate Bill 642. The tri-^encies were equally represented on the 
Steering Committee. The System Director co-chaired the Steering Committee. The employer 
survey subcontractor sat on the Steering Committee ex officio. 

Resolutions were placed before the Steering Committee by members speaking for their 
respective institutions, the site-based committees they represented, and/or ad hoc data user 
groups with special technical expertise and interest in the follow-up system’s performance. 
Every effort was made to secure unanimity on all action recommended by the Steering 
Committee. The chief concerns expressed by the Steering Committee involved the following 
issues: 

• implications of Senate Bill 642 for TCWEC’s role in the follow-up system; 

• methods for determining the degree of training-relatedness of job placements; 

• additional data sharing agreements to optimize coverage of successful outcomes; 

• procedures for supplemental follow-up to document successful outcomes for individuals 
not located through record link^es; 

• plans for regional workshops and the statewide conference; and 

• performance calculations, report and display formats best suited to convert follow-up 
data into useful information. 

The SOICC responded to the Steering Committee’s satisfaction on each issue. (Details 
are provided in this narrative, passim) By giving practitioners a voice, the governance structure 
facilitated consensus and "buy-in" to increase the likelihood that follow-up data would be Msed 
as intended to drive program improvement. 
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3) Obtaining Outcomes Data 



The primary task of the Follow-Up System in Program Year 1993-94 was to obtain 
outcome information on the former students and participants of the workforce development 
programs. (Breadth of the operation was increased to serve programs operated by TEA and 
TDoC as well as those previously served on behalf of the Coordinating Board and by 
increasing the employer follow-up survey tenfold. Such rapid increases in service levels could 
not be achieved without improving the degree of automation.) The process for obtaining 
outcomes data consists of three phases: seed record extraction, record matching, and the 
employer survey. (The process flow is depicted in Appendix I.) 

Seed Records 

To determine what data elements should be included in the seed records, each of the 
three agencies participating in the 1993-94 follow-up system developed hypotheses about the 
likely causes of potential variance in outcomes across programs or subpopulations served. 
Hypotheses were derived from agency/program mission statements, goals and objectives, 
eligibility rules, funding and program approval guidelines, and performance histories. The 
hypotheses translate notions of how the programs should have worked for various eligible 
subgroups into empirical questions. Seed records contain background information on former 
students or participants such as demographic information, program participation and/or 
completion status, and various special populations. Such items constitute the principle 
independent variables used in disaggregating and explaining variance in documented outcomes. 

While the tri-agencies share a common goal to increase the supply of skilled workers, 
they operate under different rules regarding participant eligibility and the kinds of services to 
be delivered. That is, while all three agencies wanted to document the same range of successful 
outcomes (as dependent variables), their hypotheses and research questions were slightly 
different. Moreover, the tri-agencies maintained separate information management systems with 
data elements included and defined independently of data decisions made by other workforce 
development partners. Even where conforming amendments in federal legislation or 
overlapping missions, goals and objectives focused the three agencies’ efforts in a common 
direction, relevant data items were not necessarily recorded in identical formats in the separate 
information mans^ement systems. 

In deciding how to construct the seed records, the System Director in consultation with 
the tri-agency management team had to provide sufficient flexibility to address the varied needs 
and capacities of the partners. Balanced against the flexibility principle was the need for 
efficiency. In the start-up phase, early pilot participants (NHMCC and the Brazosport Group 
for higher education, Leander ISD for public education) manually constructed seed records. 
That proved to be an inefficient, time and resourceconsuming effort. In that the variables of 
interest (i.e., those most likely to have explanatory power) are already contained in each of the 
partners’ management information systems, inclusion of items in the seed record was limited 
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to elements in existing records. That decision saved the partner a^^encies from the burden of 
manually constructing new variables. Extraction of seed records was fully automated. 

The tri-agency management team also determined that it would be inefficient to write 
separate record matching programs to link each agency’s seed records to the data bases 
containing outcomes information. A matching program consists of several decision rules: 

What item found in both the seed record and the linked files identifies a unique match? 

Where in the Seed record file layout can that element be found? Where is it in the 
linked data base? 

When a unique match is identified, what items in the linked data base should be 
written to the seed record? 

Where in the seed record file layout should the specified elements from the linked data 
base be written? 

Given these decision rules, the tri-agency management team found it could balance 
flexibility against efficiency with two rules of its own for seed record construction: 

1) Since Social Security numbers contained in both the agencies’ management information 
systems and the linked data bases provide the basis for a unique match, that variable 
should be located in the same position and in the same format in each agency’s seed 
record file layout. 

2) Seed records should be of identical length so the matching program can begin 
appending information from the linked data bases in the same position in every file 
layout. Aside from conforming to the positioning and format of the Social Security 
number, each agency was at liberty to include other items of its choice in the seed 
record in any other available position so long as the file layout did not exceed the 
specified maximum length. 

File layouts for the three agencies’ seed records and appended information appear in 
Appendix II. While there are a number of common elements, note that each agency was able 
to address questions that might not be applicable or of interest to the other partners. 

Empirical research methods often raise as many questions as they answer. Unanticipated 
results - either contrary to prevailing theory or inconsistent with prior performance - suggest 
that additional independent variables need to be explored as possible explanations. It would 
be tempting (but inefficient) to include every item from each agency’s management infor- 
mation system in the seed records it submits for follow-up. By adhering to a fixed seed record 
length of 62 bytes, the agencies were forced to be selective. 
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Some important explanatory variables, however, might be overlooked when items are 
selected for seed records. The agencies need the flexibility to look at additional independent 
variables (other than those in their seed records) as the need arises. Rather than enlarge the 
seed record to cover every possibility, the tri-agency management team determined that such 
flexibility could be preserved if, when student/participant files were returned to the 
agencies/program administrators. Social Security numbers were neither stripped nor encrypted. 
That would allow the agencies/program administrators o link the records back to their own 
management information systems to extract and append additional variables at the very end 
of the records returned. Two hypothetical cases illustrate this flexibility. 

Grading practices are locally determined. Wide variance in those practices would make 
statewide analysis of the correlation between grade point averages (GPA) and successful 
outcomes meaningless. Therefore, the tri-agency management team decided not to include 
grade information in seed records. Nonetheless, a particular school distria or college might 
want to assess the predictive power of its grading praaices. Upon return of the files, a 
program administrator could link seed records by Social Security number back to an in-house 
data base to extract or manually construct additional independent variables (such as GPA or 
GPA within major) at the end of each record for further analysis. 

Student/participant intent is undoubtedly a crucial determinant of post-program 
behavior. However, there currently is no agreed upon method for capturing "intent"; some 
service providers do not even attempt to collea student intent information. While the tri- 
agency management team recognized the potential explanatory power of student intent, they 
decided not to include the variable in seed records. Again, if program administrators choose 
to examine the correlation between student intent and outcomes, they may do so by relinking 
returned files to in-house data bases. 

Record Matching 

Each of the tri-agencies was responsible for extracting seed records for the cohorts to 
be studied. Under the 1993-94 follow-up grant, each extraaed seed records for former students 
or participants who completed or left programs during the program/school year ending July 
1, 1993. Seed records were delivered to the SOICC. 

Record matching consisted of four phases: 

• developing data sharing agreements; 

• processing by the Coordinating Board; 

• processing by TEC; and 

• conducting the employer follow-up survey. 
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Data Sharing Agreements 



Before record matching can be conducted, data sharing agreement among the tri- 
agencies and the agencies administering the linked data bases had to be in place. Where data 
sharing agreements were missing, the SOICC negotiated them. (A sample data sharing 
agreement is provided in Appendix HI. To move the system closer to the ideal, the Director 
continues to negotiate agreements for breadth of access to additional data bases likely to 
contain pertinent outcomes information.) These agreements cover basic points: 

Under what authority does the SOICC and its customers request data release? 

Under what authority is the data provider authorized to release requested information? 

For what legitimate purpose(s) will the data be used? 

What benefits will result from the analysis of the data? 

What security arrangements will be made to safeguard the released data while in the 
hands of the SOICC and its customers? 

How will reports and data displays be formatted to avoid release of individually 
identihable information? 

What penalties will be imposed if the data sharing agreement is breached? (upon the 
agency? upon the responsible individual?) 

Higher Education Match 

The SOICC first forwarded the seed records to the Coordinating Board for matching 
with the master enrollment files for the fall term, 1993. The master enrollment files contain 
information on all students pursuing higher education at any of the state’s publicly funded 
community and technical colleges, universities, or health science centers. When a match was 
identified, the following information was appended to the seed record: 

Where did the student enroll? ^>7 FICE code) 

In what kind of institution (community/technical college, university, or health 
science center)? 

What was the student’s declared major (if any)? 

Is that major considered academic, technical, or Tech Prep? 

How many credit hours were attempted in the matched term? 

In what year and semester was the match found? 

For a detailed file layout, see Appendix II. 
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TEC Match 



Once the higher education match was completed, the SOICC retrieved the files and 
delivered them to the TEC for matching against available quarterly UI wage-records. (These 
records cover approximately 97% of workers in Texas.) The TEC maintains five quarters of 
data on-line. Employers have a deadline for submitting quarterly data but may correa the 
information they submit at any time; therefore, the UI wage-records rnay be constantly 
■updated. However, the TEC releases UI wage-record information as "official” approximately 
five months after the close of each quarter. To afford ample opportunity for June 1993 
graduates/completers to search for employment, the management team wanted access to UI 
wage-records for the fourth quarter of 1993. The match run was conduaed by TEC when 
those data became available officially in May, 1994. TEC followed these decision rules in 
conducting the match: 

Match against most recent quarter (4th quarter 1993) first. 

A. If a matching record is found, append the data to the seed record for every 
employer found in that quarter. 

1) Append wage record information to the seed record only for most rece nt 
quarter where a hit was found . 

2) Can there be more than one set of employment data for an individual in the 
most recent quarter matched? YES 

3) Can there be employment data for more than one quarter appended to any seed 
record? NO 

B. If no hit is found in the 4th quarter, match in sequence against: 3rd quarter 
1993; 2nd quarter 1993; 1st quarter 1993; then 4th quarter 1992. 

When a match was found, the following information was appended to the seed record: 

What firm employed the former student/ participant? 

Where was the firm located? 

What is the principle business or industry artivity engaged in by the firm? 
How much did the former student/participant earn in the matched quarter? 
In what quarter was the most recent match found? 

For a detailed file layout, see Appendix II. 

While as many as five quarters were matched, the study was not longitudinal. Ideal 
depth of coverage will be achieved only when pre-enrollment and post-completion labor 
market statuses are compared and when outcomes are documented over multiple post- 
completion time periods. 
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Employer Survey 

Why is an Employer Survey Necessary? 

Senate Bill 642 specifies that, for depth of coverage, occupationally-specific information 
should be collected for evaluating the labor market outcomes of workforce development 
programs. While UI wage-records include the industrial classification of firms employing 
workers covered by the Texas Unemployment Compensation Act, occupational titles for indi- 
vidual employees are not included. "Enhancement” of the UI wage-record to include occupa- 
tional titles has been discussed but no decision has been reached on modifying the quarterly 
reports. Until such time as the state adopts an enhanced UI wage-record, occupationally 
specific employment information must be obtained through traditional survey techniques. 
While the employer follow-up is automated, it adds a step to the process that would not be 
necessary if the requisite data items could be extracted from enhanced UI wage-records. 

Prior to creation of the automated follow-up system, service providers conducted 
separate surveys to collect occupationally-specific information. They attempted to contact 
former students/participants at the last known address or telephone number. If contaa was 
made, they had to rely on the former student/participants’ self-reported employment out- 
comes. To verify employment, they had to obtain permission to contaa^the employer, locate 
the firm, and hope that the person contacted can and will provide accurate information. The 
method was costly and time-consuming for the service providers. In addition, employers were 
burdened with employment verification requests. They were approached separately at different 
times of the year by multiple service providers each using a different survey instrument. 



Consolidation Reduces Survey Burden 

In issuing an employer follow-up survey subcontract on behalf of the workforce 
development partners, the automated follow-up system accomplished two things: 

• it relieved the service providers of the bulk of the data collection effort; and 

• it relieved employers of excessive survey burdens. 

Rather than rely on the service providers’ records on last known point of contact for 
their former students/participants, the follow-up survey was distributed to employers at the 
address contained in TEC files. Whereas contact information in service providers’ manual files 
were seldom updated as their former students/participants changed addresses and phone 
numbers, the use of employer contact information electronically stored and updated in the 
TEC records allowed the follow-up system to automate subject location. No matter how many 
times they moved so long as they were employed by a firm covered by the Texas Unemploy- 
ment Compensation Act, subjects can be located through the UI wage-record match for 
inclusion in the employer follow-up survey. 
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Contact information for all subjects in the cohorts being studied were sorted by 
employer. Mail labels and survey instrument printing were automated. Requests for infor- 
mation on all former students/ participants (regardless of program status) employed by a single 
firm were consolidated on one survey instrument. A cover letter assured employers that they 
would be approached at only one point during the year with a single, standardized format and 
that the information they provided would be used for legitimate purposes with sufficient 
safeguards in place to protect data privacy and confidentiality. Such assurances increase the 
likelihood that employers will respond and that their responses will be accurate. 

Employers were asked to supply the following information: 

What is the former student/participant’s occupational title? 

At what worksite was the former student/participant employed? 

Did the former student/participant work full time/full quarter during the 
quarter for which earnings data were available? 



Persistence Efforts Stimulate High Response Rate 

The employer survey subcontractor persisted in follow-up to ensure high response rates. 
The subcontractor maintained an automated response log. Second and third wave reminder 
notices were generated for employers who had not responded at predetermined intervals. Non- 
responding firms employing five or more follow-up subjects were contacted by telephone and 
encouraged to respond. The employer survey subcontractor’s contact information was supplied 
on the face of the survey instrument to encourage employers to call for technical assistance 
and/or explanations of the survey’s purpose. Achieving a response rate in excess of 85% 
improves the breadth of coverage and increases the likelihood that data will be perceived as 
valid and reliable for use in program evaluation and planning. 



Coding Employer-Supplied Occupational Titles 

Employers submitted actual occupational payroll titles even if unique to the firm. The 
employer survey subcontractor was responsible for assigning standardized codes to the occupa- 
tional titles. Available tools for determining the training-relatedness of job placements rely on 
the Occupational Employment Statistic (OES). The JTPA system uses OES codes in its man- 
agement information system; the SOICC and TEC use OES codes in labor market demand 
forecasts. While other coding systems are available, the tri-agency management team agreed 
to use the OES system. More than half of the occupational titles submitted by employers 
corresponded to titles defined in the OES system. The vast majority of the remaining titles 
corresponded to definitions in the much larger Dictionary of Occupational Titles pOT). 
Employer-supplied DOT titles were converted to OES code equivalents easily by using the 
DOT-to-OES crosswalk developed by the NOICC. The SOICC also added more than 20,000 
payroll titles from responses on IRS forms to the OES autocoding system. 
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Where employer-supplied titles could not be found in either the OES or DOT systems, 
or in the IRS/census, other techniques were used. Employers were recofttacted and asM to 
provide more detail about the duties and tasks performed under titles not yet coded. By 
matching duties and tasks to code definitions, employer-supplied titles were converted to their 
OES equivalents. Dr. Jim Reed had conducted similar research for several years prior to his 
selection as the employer survey subcontraaor. Over time he constructed history mes for 
unique employer-supplied titles. In addition, the Florida follow-up system use^!^ identic^ 
technique to build history files. By sharing resource, both Dr. Rwd and FETTIP accessed 
information that expanded coverage of the automated coding technique. They also decreased 
their need for recontacts since most employer-supplied unique titles have appeared in prior 
studies and appropriate OES code conversions have been entered into one or the other s 
history files. Less than 5% of the employer-supplied occupational titles remained uncodeable 
after all contingency techniques were exhausted. The OES codes assigned to the employer- 
supplied occupational titles were included in the files returned to service providers to improve 

their usefulness. 

Converting Zip Code into Useful V^ork-Site Infomtution 

The firm address contained in the UI wage records might not represent the work-site. 
Rather, it may be the firm’s headquarters, the location of its payroll processing operations, or 
the location of an out-source payroll processing firm. Therefore, the survey instrument asked 
for the Zip code of the work-site location. Such information allows researchers to study the 
migration patterns of former students/participants as they search for employment. Information 
by Zip code can be aggregated into other meaningful geographic units. The employer suiwey 
subcontractor converted employer supplied Zip codes into city, state and county information. 
To improve the usefulness of the data files, the SOICC generated a crosswalk which c«3nverts 
FIPS (county codes) into Quality Work Force Planning regions, JTPA SDAs, the Comp- 
troller’s Uniform State Service Regions (USSRs), TEC Regions. 



Identifying Subjects Who Worked Full Time/Full Quarter 



UI wage records contain total earnings for each quarter worked. They do not contain 
information about the hourly wage or hours worked during the quarter for each employee. 
Inclusion of either variable has been proposed as part of the enhanced UI wage-r^ord. 
However, no decision has been made about modifying the quarterly reports. Until the UI 
wage-record is enhanced, other methods must be used to make total quarterly earnings more 

meaningful. 



At the beginning of Program Year 1992-93, the follow-up Steering Committ^ debated 
the feasibility of various approaches for obtaining more detailed information. The survey 
instrument could ask for hourly wage or hours worked. The Committee was advised by Jay 
Pfeiffer, the director of the Florida follow-up system, that inclusion of such a question would 
require firms to have both their personnel and payroll departments to «spond to separate 
portions of the instrument. That cuts the likelihood of response m half. Mr. Pfeiffer advised 
that asking employers to identify subjects who worked full time/full quarter would increase 
response rates while rendering information that was only slightly less detailed. (His advice was 
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based on experience. When FETPIP asked for hours worked in their first survey, they 
achieved a 40% response rate; since converting to the full time/full quarter flag, they have 
consistently achieved 80% response rates or better.) To confirm Mr. Pfeiffer’s advice, three 
drafts of the survey instrument were submitted to the regional Job Service Employers’ 
Committees QSEC). They confirmed that employers would be more likely to respond to the 
form requesting only the full time/full quarter flag. The simpler survey instrument, therefore, 
was adopted. That decision increased the breadth of coverage and increased the likelihood that 
follow-up data would be sufficiently valid and reliable for use in program planning and 
evaluation. 

4) Analyzing Results 

Because community and technical colleges have the longest history of participation in 
automated follow-up. Coordinating Board staff under the direction of Helen Giraitis provided 
analysis of follow-up data for the cohorts of former higher education students. Because the 
JTPA system has used follow-up data collected through participant telephone surveys by Texas 
A&M University under contract to TDoC, staff members in the Workforce Development 
Division were responsible for analyzing automated follow-up data on former Title IIA and 
Title III participants. 0im Gaston and Robin Campbell led the TDoC data analysis team.) 
Marc Anderberg, Follow-Up System Director, with the assistance of pilot district liaison as 
well as Dr. Mark Butler, Dave Kinnaman and Judith hetherly of TEA’s Career and Tech- 
nology Education Division took responsibility for the analysis of follow-up data for the public 
education pilots. John Syers (Region IV ESC), Dr. Larry Kohler and Leonard Thielin 
(McAllen ISD), and Dr. Sandra Neubert (Waco ISD) were particularly helpful in analyzing 
public education outcomes. The division of labor improved system efficiency. It ensured that 
experienced and respected higher education and JTPA practitioners analyzed and promoted 
the use of their own data. It concentrated technical assistance for public education where it was 
most needed. By engaging in a collaborative process, it enhanced the buy-in of public educa- 
tion practitioners. 

Results for Public Education 

Pilot distrias were selected from volunteers. Their graduates were not representative 
of the entire state’s high school completers. Therefore, it would be inappropriate to combine 
the files of the pilot schools to develop a composite picture. Analysis herein is limited to 
separate reports on each pilot. 

First a prototype report card was developed for each pilot district. The labor market 
outcomes and continued pursuit of education are displayed on facing pages. Labor market 
outcomes are broken down further to show students the likelihood of earning more than the 
equivalent of minimum wage if they seek employment immediately upon receiving a high 
school diploma. Data also were analyzed to determine the types of employment high school 
graduates are most likely to receive. 
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ALDINE INDEPENDENT SCHOOL DISTRICT 
1992-93 Graduating Cohort 

(all campuses/studcnt with some vocational training fall term) 



Number of usable records 


651 


Successful Outcomes Documented 


Working and/or pursuing higher education 


527 


(public institution) in Texas 


81.0% 


Not Located 


Not located via record link^e techniques 


124 


to available data bases 


19.0% 



Labor Market Outcomes 
(post-graduation through fourth quarter 1993) 

Employed in Texas in job covered by 
state's Unemployment Compensation Act 

Woiking and enrolled in Texas 
public higher ed. institution 

as %-age of those working 

Working only/not enrolled in Texas 
public higher ed. institution 

as %-age of those working 

Earning more than $2,205/quarter 
(min. wage X 40 hrs. X 13 weeks) 
as %-age of those working 

Employed in the federal civil service 

Employed by the U.S. Postal Service 

Entered the Military 



444 

68*27o 

201 

30.9% 

45.3% 

243 

37.3% 

54.7% 

115 

17.77o 

25.97o 

to be inserted here 
to be inserted here 
to be inserted here 



Top industries by number of job placements for Aldine grads: 



SIC 


Industry 


Placements 


% of grads 


54 


Grocery Stores 


79 


12.1% 


58 


Eating & Drinking Est. 


69 


10.6% 


73 


Business Services 


67 


10.3% 


53 


General Merchandise Stores 


34 


5.2% 


55 


Auto Service Stations 


20 


3.0% 


56 


Apparel Stores 


20 


3.0% 


80 


Health Services 


16 


2.5% 



% of working grads 
17.8% 
15.5% 
15.17o 
77% 

4.57o 

4.5% 

3.67o 



Seven industries account for 68*77o working Aldine 
grads located in Texas jobs covered by state unemploy- 
ment compensation act« 
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ALDINE (Continued) 



Pursuing Higher Education 

284 

43 . 6 % 



Enrolled in Texas public institution 
of higher ed. (fall term, 1993) 



Enrolled in Tx public higher ed 201 

institution and working in Texas 30.9% 

as %-age of those enrolled 70.8% 

Enrolled in Tx public higher ed 83 

institution but not found working in Texas 12.7% 

as %-age of those enrolled 29.2% 

Enrolled in Tx public community or 183 

technical college 28.1% 

as %*age of those enrolled 64.4% 

Enrolled in Tx public senior insti- 101 

tution (university) 15.5% 

as %-age of those enrolled 35.6% 

Enrolled in Tx public health sciences 0 

institution 0% 

as %-age of those enrolled 0% 



Top Texas public institutions of higher education by placements of Aldine grads: 





# of erads enrolled 




as % of Aldine grads pursuing 


Institution 


as % of all grads 


hieher ed at Tx public institution 


N. Harris/Montgomery CC 159 


24.4% 


56.0% 


Sam Houston State 


19 


2.9% 


6.7% 


Prairie View A&M 


14 


2.2% 


4.9% 


U of Houston 


13 


2.0% 


4.6% 


U of Houston - Downtown 13 


2.0% 


4.6% 


Steven F. Austin U 


13 


2.0% 


4.6% 




Six Texas public institutions account for 81.4% 

of former Aldine grads found pursuing higher education 




m a Texas public institution. 




Top five declared majors among Aldine grads pursuing higher education: 


Maior area 


# of erads enrolled 




as % of Aldine grads pursuing 


as % of all erads 


higher ed at Tx public institution 


Health Sciences 


50 


20.2% 


17.6% 


Business 


39 


6.0% 


13.7% 


Protective Services 


16 


2.5% 


5.6% 


Education Services 


14 


2.2% 


4.9% 


Liberal Arts 


12 


1.8% 


4.2% 



^ m rnmm ^ ^ w* « 

declared by former Aldine grad pursuing higher ed in 
Texas public institution. 



Undeclared 



72 



11 . 0 % 



25 . 4 % 
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AMARILLO INDEPENDENT SCHOOL DISTRICT 
1992-93 Graduating Cohort 

(all campuses/all students with some vocational training fall term) 



Number of usable records 



Successful Outcomes Documented 

Working and/or pursuing higher education 627 

(public institution) in Texas 87.3% 

Not Located 

Not located via record linkage techniques 91 

to available data bases 12.7% 



Labor Market Outcomes 

(post-graduation through fourth quarter 1993) 

Employed in Texas in job covered by 
state’s Unemployment Compensation Act 

Working and enrolled in Texas 
public higher ed. institution 

as %-age of those working 

Working only/ not enrolled in Texas 
public higher ed. institution 

as %-age of those working 

Earning more than $2,205/quarter 
(min. wage X 40 hrs. X 13 weeks) 
as %-age of those working 

Employed in the federal civil service 

Employed by the U.S. Postal Service 

Entered the Military 



568 

79.1% 



284 

39.6% 

50.0% 



284 

39.6% 

50.0% 



140 

19.5% 

24.6% 

to be inserted here 
to be inserted here 
to be inserted here 



Top industries by number of job placemenU for Amarillo grads 



SIC Industry 

58 Eating & Drinking Est. 

54 Grocery Stores 

53 General Merchandise Stores 

80 Health Services 

59 Miscellaneous Retail Stores 

56 Apparel Stores 

73 Business Services 



Placements 
144 
52 
41 
40 
37 
32 
23 



% of grads 
20 . 1 % 
7.2% 
5.7% 
5.6% 
5.2% 
4.5% 
3.2%. 



% of working grads 
25.4% 

9.2% 

7.2% 

7.0% 

6.5% 

5.6% 

4.0% 



Seven industries account for 64.9% working Amarillo 
grads located in Texas jobs covered by state unemploy- 
ment compensation act. 
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AMARILLO (Continued) 



Pursuing Higher Education 



Enrolled in Texas public institution 343 

of higher ed. (fall term, 1993) 56.8% 




Enrolled in Tx public higher ed 
institution and working in Texas 
as %-age of those enrolled 


284 

39.6% 

82.8% 


Enrolled in Tx public higher ed 
institution but not found working in Texas 
as %>age of those enrolled 


59 

8.2% 

17.2% 


Enrolled in Tx public community or 
technical college 

as %-age of those enrolled 


251 

35.0% 

73.2% 


Enrolled in Tx public senior insti- 
tution (tmiversity) 

as %*age of those enrolled 


92 

12.8% 

26.8% 


Enrolled in Tx public health sciences 
institution 

as %-age of those enrolled 


0 

0% 

0% 


Top Texas public institutions of higher education by placements of Amarillo grads: 

as % of Amarillo grads pursuing 

Institution # of grads enrolled as % of all grads higher ed at Tx public institution 


Amarillo College 232 32.3% 67.6% 

West Texas A&M 49 6.8% 14.3% 

Texas Tech 21 2.9% 6.1% 

UT - Austin 7 1.0% 2.0% 

Four Texas public institutions account for 90*0% 



of former AmarQlo grads found pursuing higher education 
in a Texas public institution. 



Top five declared 

Major area 
Health Sciences 
Business 
Liberal Arts 
Psychology 



majors among Amarillo grads pursuing higher education: 

as % of Amarillo grads pursuing 
# of grads enrolled as % of all grads higher ed at Tx public institution 



89 


12.4% 


25.9% 


46 


6.4% 


13.4% 


28 


3.9% 


8.2% 


17 


2.4% 


5.0% 



Four major areas of study account for 52.5% of majors 
declared by former Amarillo grad pursuing higher ed in 
Texas public institution. 



Undeclared 



73 



10 . 2 % 



21.3% 
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BEAUMONT INDEPENDENT SCHOOL DISTRICT 



1992-93 Graduating Cohort 

(all campuses/all students with some vocational training fall term) 


Number of usable records 


332 




Successful Outcomes Documented 

Working and/or pursuing higher education 
(public institution) in Texas 


270 

81.3% 




Not Located 

Not located via record linkage techniques 
to available data bases 


62 

18.7% 




Labor Market Outcomes 
(post-graduation through fourth quarter 1993) 






Employed in Texas in job covered by 
state’s Unemployment Compensation Act 


237 

71.4% 




Working and enrolled in Texas 
public higher ed. institution 

as %-age of those working 




108 

32.5% 

45.6% 


Working only/not enrolled in Texas 
public higher ed. institution 

as %-age of those working 




129 

38.9% 

54.4% 


Earning more than $2,205/ quarter 
(min. wage X 40 hrs. X 13 weeks) 
as %-age of those working 




32 ■ 

9.6% 

13.5% 


Employed in the federal civil service 




to be inserted here 


Employed by the U.S. Postal Service 




to be inserted here 


Entered the Military 




to be inserted here 



Top industries by number of job placements for Beaumont grads: 



SIC Industry 

58 Eating & Drinking Est. 

54 Grocery Stores 

56 Apparel Stores 

53 General Merchandise Stores 

59 Miscellaneous Retail Stores 



Placements 
84 
35 
21 
20 
15 



% of grads 
25.3% 
10.5% 
6.3% 
6 . 0 % 
4.5% 



I of working grads 
35.4% 
14.8% 

8.7% 

8.4% 

6.3% 



Five industries account for 73.6% working Beaumont 
grads located in Texas jobs covered by state unemploy- 
ment compensation act. 
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BEAUMONT (Contimied) 



Pursumg Higher Education 

141 
42.5% 



Enrolled In Texas public institution 
of higher ed. (fall term» 1993) 



Enrolled in Tx public higher ed 108 

institution and working in Texas 32.5% 

as %-age of those enrolled 76.6% 

Enrolled in Tx public higher ed 33 

institution but not found working in Texas 9.9% 

as %-age of those enrolled 23.4% 

Enrolled in Tx public community or 21 

technical college 6.3% 

as %-age of those enrolled 14.9% 

Enrolled in Tx public senior insti- 119 

tution (tmiversity) 35.8% 

as %-age of those enrolled 84.4% 

Enrolled in Tx public health sciences 1 

institution 0.3% 

as %-age of those enrolled 0.7% 



Top Texas public institutions of higher education by placements of Beaumont grads: 



Institution 


# of grads enrolled 


as % of all grads 


higher ed at Tx public i 


Lamar University 


81 


24.4% 




57.4% 


Prairie View A&M 


17 


5.1% 




12.1% 


Lamar Inst, of Tech. 


12 


3.6% 




8.5% 


UT - Austin 


7 


2.1% 




5.0% 




Six Texas public institutions account for 


83.0% 



of former Beaumont grads found pursuing higher education 
in a Texas public institution. 



Top five declared majors among Beaumont grads pursuing higher education: 

as % of Beaumont grads pursuing 
# of grads .enrolled as % of all grads hi gher ed at Tx public institution 



Major area 
Health Sciences 
liberal Arts 
Multi-Disciplinary 
Business 



18 

8 

6 

4 



5.4% 

2.4% 

1 . 8 % 

1 . 2 % 



12 . 8 % 

5.6% 

4.2% 

2 . 8 % 



Five major areas of study account for 79.9% of majors 
declared by former Beaumont grad pursuing higher ed in 
Texas public institution. 



Undeclared 



76 



22.9% 



53.9% 



Final Report: Automated Student and Adult Learner Follow-Up, 1993*94 

O 

ERIC 



39 



Page 31 



HEREFORD INDEPENDENT SCHOOL DISTRICT 
1992-93 Graduating Cohort 
(all students with some vocational training fall term) 



Number of usable records 



175 



Successful Outcomes Documented 

Working and/or pursuing higher education 
(public institution) in Texas 

Not Located 

Not located via record linkage techniques 
to available data bases 

Labor Market Outcomes 
(post-graduation through fourth quarter 1993) 

Employed in Texas in job covered by 
state’s Unemployment Compensation Act 

Working and enrolled in Texas 
public higher ed. institution 

as %-age of those working 

Working only/not enrolled in Texas 
public higher ed. institution 

as %-age of those working 

Earning more than $2,205/ quarter 
(min. wage X 40 hrs. X 13 weeks) 
as %-age of those working 

Employed in the federal civil service 

Employed by the U.S. Postal Service 



149 

85.17o 



26 

14.9% 



141 

80.6% 



55 

31.4% 



39.0% 



86 

49.1% 



61.0% 



33 

18.9% 



23.4% 



to be inserted here 
to be inserted here 



Entered the Military 

Top industries by number of job placements for Hereford grads: 


to be inserted 1 


SIC 


Industry 


Placements 


% of grads 


% of working i 


58 


Eating & Drinking Est. 


30 


17.1% 


21.3% 


20 


Food & Kindred Products 


15 


8.6% 


10.6% 


54 


Grocery Stores 


11 


6.3% 


7.8% 


82 


Education Services 


10 


5.7% 


7.1% 


53 


General Merchandise Stores 


8 


4.6% 


5.7% 


51 


Wholesale - Nondurable Goods 


6 


3.4% 


4.3% 


01 


Agriculture (crops) 


6 


3.4% 


4.3% 


80 


Health Services 


5 


2.9% 


3.5% 



grads located in Texas jobs covered by state unemploy- 
ment compensation act. 
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HEREFORD (Continued) 



Pursuing Higher Educs*^ion 



Enrolled in Texas public institution 63 

of higher ed. (fall term, 1993) 72.9% 



Enrolled in Tx public higher ed 55 

institution and working in Texas 31.4% 

as %-age of those enrolled 87.3% 

Enrolled in Tx public higher ed g 

institution but not found working in Texas 4 . 6 % 

as %-age of those enrolled 12 . 7 % 

Enrolled in Tx public community or 15 

technical college g,g% 

as %-age of those enrolled 23.8% 

Enrolled in Tx public senior insti- 4 g 

tution (university) 27.4% 

as %-age of those enrolled 76.2% 

Enrolled in Tx public health sciences 0 

institution 0 % 

as %-age of those enrolled 0 % 



Top Texas public institutions of higher education by placements of Hereford grads: 

as % of hereford grads pursuing 

# of grads enrolled as % of all grads 
32 18.3% 

8 4.6% 

6 3.4% 

6 3.4% 

6 3.4% 

Five Texas public institutions account for 92.0% 

of former Hereford grads found pursuing higher education 
in a Texas public institution. 



Institution 
West Texas A&M 
Amarillo College 
Texas Tech 
Angelo State 
South Plains College 



higher ed at Tx public institution 
50.8% 

12.7% 

9.5% 

9.5% 

9.5% 



Top six declared majors among Hereford grads pursuing higher education: 

as % of Hereford grads pursuing 

as % of all grads 
13.7% 

6.3% 

2.9% 

1.7% 

1.7% 

1.7% 



Major area 
Liberal Arts 
Health Sciences 
Business 

Multi-Disciplinary 
Life Sciences 
Agribusiness 



# of grads enrolled 

24 

11 



Undeclared 



higher ed at Tx public institution 
38.1% 

17.5% 

7.9% 

4.8% 

4.8% 

4.8% 

Six major areas of study account for 77.9% of majors 
declared by former Hereford grad pursuing higher ed in 
Texas public institution. 

1.1% 3 . 2 % 
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LEANDER HIGH SCHOOL 
1992*93 Graduating Cohort 
(all students with some vocational training fall term) 



Number of usable records 




102 




Successful Outcomes Documented 

Working and/or pursuing higher education 
(public institution) in Texas 




99 

97.1% 




Not Located 

Not located via record linkage techniques 
to available data bases 




3 

2.9% 




Labor Market Outcomes 
(post-graduation through fourth quarter 1993) 








Employed in Texas in job covered by 
state’s Unemployment Compensation Act 




89 

87% 




Working and enrolled in Texas 
public higher ed. institution 

as %-age of those working 






48 

47.1% 

54% 


Working only/not enrolled in Texas 
public higher ed. institution 

as %-age of those working 




41 

40.2% 

46% 


Earning more than $2,205/ quarter 
(min. wage X 40 hrs. X 13 weeks) 
as %-age of those working 






23 

22.5% 

26% 


Employed in the federal civil service 






to be inserted here 


Employed by the U.S. Postal Service 






to be inserted here 


Entered the Military 






to be inserted here 


Top industries by number of job placements: 

Sig Industry Placements % of grads % of workine erads 

58 Eating/Drinking Est. 13 12.7% 14.6% 

54 Grocery Stores 11 10.8% 12.4% 

73 Business Services (temps) 10 9.8% 11.3% 

50 Wholesale - Durable 6 5.9% 6.7% 

53 General Merchandise Stores 5 4.9% 

Five industries account for 50.6% of working Leander 
grads located in Texas jobs covered by state unemploy- 



ment compensation act. 
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LEANDER (Continued) 



Pursuing Higher Education 



.Z' 



Enrolled in Texas public institution 
of higher ed. (fall term, 1993) 

Enrolled in Tx public higher ed 
institution and working in Texas 
as %-age of those enrolled 

Enrolled in Tx public higher ed 
institution but not found working in Texas 
as %-age of those enrolled 

Enrolled in Tx public community or 
technical college 

as %-age of those enrolled 

Enrolled in Tx public senior insti- 
tution (university) 

as %-age of those enrolled 

Enrolled in Tx public health sciences 
institution 

as %-age of those enrolled 



51 

50% 



48 

47.1% 



3 

2.9% 



30 

29.4% 



21 

20 . 6 % 



0 

0 % 



94.1% 



5.9% 



58.8% 



41.2% 



0 . 0 % 



Top Texas public institutions of higher education by placements of Leander grads: 

as % of Leander grads pursuing 



Institution 


# of erads enrolled as % 


of all grads 


higher ed at Tx public institution 


Austin Com. College 


27 


26.5% 


53.0% 


U.T. - Austin 


9 


8.8% 


17.6% 


Southwest Tx State 


3 


2.9% 


5.9% 


Steven F. Austin U. 


3 


2.9% 


5.9% 


Tarleton State U. 


3 2.9% 5.9% 

Five Tx public institutions of higher ed account for 88.3% 
of former Leander grads found pursuing higher education in Texas 
public institution. 


Top Hve declared majors among Leander grads pursuing higher education: 

as % of Leander grads pursuing 


Maior area 


# of erads enrolled as % of all erads 


hieher ed at Tx Dublic institution 


Liberal Arts 


24 


23.5% 


47.1% 


Business 


6 


5.9% 


12.0% 


Health Professions 


5 


4.9% 


10.0% 


Ag (crop) 


3 


3.0% 


5.9% 


Ag (animal) 


3 


3.0% 


5.9% 



Undeclared 



Five major areas of study account for 80.9% of majors 
declared by former Leander grad pursuing higher ed in 
Texas public institution. 

0 % 



o 

ERIC 
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M'ALLEN INDEPENDENT SCHOOL DISTRICT 
1992*93 Graduating Cohort 

(all campuses/all students with some vocational training fall term) 



Number of usable records 

Successful Outcomes Documented 

Working and/or pursuing higher education 
(public institution) in Texas 

Not Located 

Not located via record linkage techniques 
to available dau bases 



765 



596 

77.9% 



169 

22 . 1 % 



Labor Market Outcomes 
^st-graduation through fourth quarter 1993) 



Employed in Texas in job covered by 




445 




state’s Unemployment Compensation Act 




58.2% 




Working and enrolled in Texas 






230 


public higher ed. institution 






30.1% 


as %*age of those working 






51.7% 


Working only/not enrolled in Texas 






215 


public higher ed. institution 






28.1% 


as %-age of those working 






48.3% 


Earning more than $2,205/quarter 






67 


(min. wage X 40 hrs. X 13 weeks) 






8.8% 


as %-age of those working 






15.1% 


Employed in the federal civil service 






to be inserted here 


Employed by the U.S. Postal Service 






to be inserted here 


Entered the Military 






to be inserted here 


Top industries by number of job placements for McAllen grads: 
SIC Industry Placements % of erads 


% of workine grads 


54 Grocery Stores 


68 


8.9% 


15.3% 


53 General Merchandise Stores 


65 


8.5% 


14.6% 


58 Eating & Drinking Estab. 


63 


8.2% 


14.2% 


56 Apparel Stores 


56 


7.3% 


12.6% 


59 Miscellaneous Retail 


32 


4.2% 


7.2% 


73 Business Services (temps) 


25 


3.3% 


5.6% 


82 Education Services 


22 


2.9% 


4.9% 


80 Health Services 


16 


2.0% 


3.6% 


50 Wholesale - Durable Goods 


16 


2.0% 








Nine industi'ies account for 81.6% working 






M‘AUen grads located in Texas jobs covered by 
state unemployment compensation act. 



O 

ERIC 
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McALLEN (Continued) 



Pursuing Higher Education 

Enrolled in Texas public institution 381 



of higher ed. (fall term, 1993) 49.8% 

Enrolled in Tx public higher ed 230 

institution and working in Texas 30.1% 

as %-age of those enrolled 60.4% 

Enrolled in Tx public higher ed 151 

institution but not found working in Texas 19.7% 

as %-age of those enrolled 39.6% 

Enrolled in Tx public community or 40 

technical college 5.2% 

as %-age of those enrolled 10.5% 

Enrolled in Tx public senior insti- 338 

tution (university) 44.2% 

as %-age of those enrolled 88.7% 

Enrolled in Tx public health sciences 3 

institution 0.4% 

as %-age of those enrolled 0.8% 



Top Texas public institutions of higher education by placements of McAllen grads: 

as % of McAllen grads pursuing 

Institution # of grads enrolled as % of all grads higher ed at Tx public institution 



UT - Pan American 


237 


31.0% 


62.2% 


So Tx Community College 


25 


3.4% 


6.6% 


Texas A&M 


24 


3.1% 


6.3% 


UT - Austin 


23 


3.0% 


6.0% 


UT - San Antonio 


12 


1.6% 


3.1% 


Tx A&M - Kingsville 


11 


1.4% 

Six Texas public institutions account for 


2.9% 

87.1% 



of former McAllen grads found pursuing higher education in 
a Texas public institution. 



Top five declared majors among McAllen grads pursuing higher education: 

as % of McAllen grads pursuing 



Maior area 


# of grads enrolled 


as % of all grads hiehcr cd at Tx public institution 


Bttsiness 


65 


8.5% 


17.1% 


Health Sciences 


51 


6.7% 


13.9% 


Engineering 


31 


4.1% 


8.1% 


Multi-Disciplinary 


26 


3.4% 


6.8% 


Law/Prc-Law 


18 


2.4% 


4.7% 


liberal Arts 


17 


2.2% 4.5% 

Five major areas oi study account for 55.1% of majors 
declared by former McAllen grad pursuing higher ed in 
Texas public institution. 


Undeclared 


69 


9.0% 


18.1% 



O 

ERIC 
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SILSBEE INDEPENDENT SCHOOL DISTRICT 
1992-93 Graduating Cohort 
(all students with some vocational training fall term) 



Number of usable records 




Successful Outcomes Documented 


Working and/or pursuing higher education 


72 


(public institution) in Texas 


75.0% 


Not Located 




Not located via record linkage techniques 


24 


to available data bases 


25.0% 



Labor Market Outcomes 
(post-graduation through fourth quarter 1993) 

Employed in Texas in job covered by 
state’s Unemployment Compensation Act 



67 

69.8% 



Working and enrolled in Texas 
public higher ed. institution 

as %-age of those working 



21 

22 . 0 % 

31.3% 



Working only/not enrolled in Texas 
public higher ed. institution 

as %-age of those working 



46 

47.9% 

68.7% 



Earning more than $2,205/quarter 
(min. wage X 40 hrs. X 13 weeks) 
as %-age of those working 



18 

18.8% 

26.9% 



Employed in the federal civil service 



to be inserted here 



Employed by the U.S. Postal Service 



to be inserted here 



Entered the Military 



to be inserted here 



Top 4 industries by number of job placements for Silsbee grads: 



SIC 


Industry 


Placements 


% of grads 


% of working 


54 


Grocery Stores 


19 


19.8% 


28.4% 


58 


Eating & Drinking Estab. 


12 


12.5% 


17.9% 


53 


General Merchandise Stores 


9 


9.4% 


13.4% 


24 


Lumber & Wood Products 


5 


5.2% 


7.5% 



Four industries account for 67.2% working Silsbee 
grads located in Texas jobs covered by state unemploy- 
ment compensation act. 
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SILSBEE (Continued) 



Pursuing Higher Education 

Enrolled in Texas public institution 26 

of higher ed. (fall term, 1993) 27.1% 



Enrolled in Tx public higher ed 


21 


institution and working in Texas 


21.9% 


as %-age of those enrolled 


80.8% 


Enrolled in Tx public higher ed 


5 


institution but not found working in Texas 


5.2% 


as %-age of those enrolled 


19.2% 


Enrolled in Tx public community or 


10 


technical college 


10.4% 


as %-age of those enrolled 


38.5% 


Enrolled in Tx public senior insti- 


15 


tution (university) 


15.6% 


as %-age of those enrolled 


57.7% 


Enrolled in Tx public health sciences 


1 


institution 


1.0% 


as %-age of those enrolled 


3.8% 



Top Texas public institutions of higher education by placements of Silsbee grads: 

as % of Silsbee grads pursuing 

Institution # of grads enrolled as % of all grads higher ed at Tx public institution 

Lamar University 11 11.5% 42.3% 

Sam Houston State 4 4.2% 15.4% 

Lamar Institute of Tech. 3 3.1% 11.5% 

Three Texas public institutions account for 69.2% 

of former Silsbee grads found pursuing higher education 
in a Texas public institution. 



Top five declared majors among Silsbee grads pursuing higher education: 

as % of Silsbee grads pursuing 

Major area # of grads enrolled as % of all grads hi gher ed at Tx public institution 



43 


4 


4.2% 


15.3% 


Health Sciences 


3 


3.1% 


11.5% 


Business 


3 


3.1% 


11.5% 



Three major areas of study account for 38.3% 
of majors declared by former Silsbee grads 
pursuing higher ed in Texas public institution. 



Undeclared 7 7.3% 26.9% 
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WACO INDEPENDENT SCHOOL DISTRICT 
1992-93 Graduating Cohort 

(all campu$e$/all students receiving some vocational training fall term) 



Number of usable records 




Successful Outcomes Documented 




Working and/or piusuing higher education 


187 


(public institution) in Texas 


79.6% 


Not Located 




Not located via record linkage techniques 


48 


to available data bases 


20.4% 



Labor Market Outcomes 
(post-gradiution through fourth quarter 1993) 



Employed in Texas in job covered by 165 

staters Unemployment Compensation Act 70*2% 



Working and enrolled in Texas 
public higher ed. institution 

as %-age of those working 



79 

33.6% 

47.9% 



Working only/not enrolled in Texas 
public higher ed. institution 

as %-age of those working 



86 

36.6% 

52.1% 



Earning more than $2,205/quarter 
(min. wage X 40 hrs. X 13 weeks) 
as %-age of those working 



31 

13.2% 

18.9% 



Employed in the federal civil service 



to be inserted here 



Employed by the U.S. Postal Service 



to be inserted here 



Entered the Military 



to be inserted here 



Top industries by number of job placements for Waco grads: 



SIC 


Industry 


Placements 


% of grads 


% of working grads 


58 


Eating 8c Drinking Est. 


49 


20.9% 


30.0% 


54 


Grocery Stores 


17 


7.2% 


10.3% 


56 


Apparel Stores 


16 


6.8% 


9.7% 


53 


General Merchandise Stores 


14 


6.0% 


8.5% 


73 


Business Services 


14 


6.0% 


8.5% 


59 


Miscellaneous Retail Stores 


12 


5.1% 


6.7% 



Six industries account for 73.7% working Waco 
grads located in Texas jobs covered by state unemploy- 
ment compensation act. 
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O 

ERIC 



48 



WACO (CMlioiicd) 

Purmisig Higher Education 



Enrolled in Texas public institution 101 

of higher ed. (fall term, 1993) 43.0% 

Enrolled in Tx public higher ed 79 

institution and working in Texas 33.6% 

as %-agc of those enrolled 78.2% 

Enrolled in Tx public higher ed 22 

institution but not foimd working in Texas 9.4% 

as %-age of those enrolled 21.8% 

Enrolled in Tx public community or 79 

technical college 33.6% 

as %-age of those enrolled 78.2% 

Enrolled in Tx public senior insti* 22 

tution (university) 9.4% 

as %-age of those enrolled 21.8% 

Enrolled in Tx public health sciences 0 

institution 0% 

as %-age of those enrolled 0% 



Top Texas public institutions of higher education by placements of Waco grads: 

as % of Waco grads pursuing 

Institution # of grads enrolled as % of all grads higher ed at Tx public institution 



McLennan Com. College 


57 


24.2% 


56.4% 


TSTC • Waco 


12 


5.1% 


11.9% 


Tyler Jr. College 


6 


2.6% 


6.0% 


Texas A&M 


6 


2.6% 


6.0% 


UT - Atistin 


5 2.1% 

Five Texas public institutions account for 


5.0% 

85.3% 



of former Waco grads found pursuing higher education in 
a Texas public institution. 



Top five declared majors among Waco grads pursuing higher education: 

as % of Waco grads pursuing 

Major area # of grads enrolled as % of all grads higher ed at Tx public institution 



liberal Arts 


29 


12.3% 


29.0% 


Health Sciences 


14 


6.0% 


14.0% 


Business 


11 


4.7% 


10.9% 


Psychology 


6 


2.6% 


6.0% 


Engineering 


5 


2.1% 


5.0% 



Five major areas of study account for 64.9% of majors 
declared by former Waco grad pursuing higher ed in 
Texas public institution. 



(Xiladcclsrcd 



6 



2 . 6 % 



6 . 0 % 
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The record linkage technique which has proved iuccesshil for higher education in Texas 
(since 1990) and for education and training providers in Florida (since 1985) also worked to 
document results for the Texas public education pilots in Program Year 1993-94. A total of 
3,079 useable records were obtained for the eight pilots. The percent^e of successful 
outcomes ranged from a low of 75% to a high of 97.1%. Over all, successful outcomes were 
found for 82.1% of the former public education students across eight pilot districts. 

Labor market results were fairly consistent across the pilots. Those results also were 
consistent with findings in other states (e.g., Florida) and with the observations and opinions 
voiced in the theoretic literature. This would lead researchers to hypothesize that comparable 
results would be achieved in other Texas school districts. 

lABOR MARKET OUTCOMES 



■ 


Employed 
in Texas 


Working & 
Continuing 
to Pursue 
Higher Ed 


Working 

Only 


Earnings 
> Than 
$2,205 
in Report 
Quarter 


Working 
in Eat. 
& Drink. 
Estab. 


Working 

in 

Grocery 

Stores 


%-age of 1 
Top Five 1 
Industries B 
(by Place- 1 
ments) in 1 
Service I 
Sector 1 


Low 


68.2% 


22.0% 


28.1% 


8.8% 


8.2% 


6.3% 


100% I 


1 Median 


70.0% 


33.3% 


36.7% 


14.9% 


15.1% 


9.5% 


N.A. 1 


I 


87.0% 


47.1% 


49.1% 


22.5% 


25.3% 


19.8% 


80% 1 



EDUCATION OUTCOMES 





Enrolled 
in Texas 
Publicly 
Funded 
Higher Ed 
Institution 


Working ic 
Continuing 
to Pursue 
Higher Ed 


Pursuing 

Higher 

Education 

Only 


Attending 

Texas 

Public 

Com/Tech 

College 


Attending 

Texas 

Public 

University 


Of those 1 
Pursuing | 
Higher 1 

%-age 1 

Undeclared 1 
Major 1 


Low 


27.1% 


22.0% 


2.9% 


5.2% 


9.4% 


0.0% 


Median 


45.1% 


33.3% 


11.8% 


20.4% 


24.6% 


16.8% 


1 High 


72.9% 


47.1% 


19.7% 


35.0% 


44.2% 


53.9% 
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The widest variance among labor market outcomes for the eight pilots was seen in the 
percentage of graduates who worked while pursuing higher education. That result probably 
has less to do with the quality of public education the graduates received than it does with 
regional economic differences. Simple rank order correlation coefficients suggest that economic 
determinants be explored more extensively. Statewide conclusions can not be drawn from a 
small number of non-representative districts. Nonetheless, the rank order correlation coeffi- 
cients computed below were significant at least at the .100 level^ for the six districts lying 
within Metropolitan Statistical Areas* (MSAs). 

Graduates’ ability to finance their continued pursuit of education may depend in large 
part on family income. Seed records did not contain data on individual’s family income. (Such 
data might be available in local data bases and could be retrieved by program administrators 
for detailed analysis. The next best source of relevant data would be per capita income or 
median household income for the census tracts served by each district or campus.^ In lieu of 
individual family income data, 1990 DHS estimates of the incidence of poverty for the regions 
served by each pilot district were ranked in ascending order and compared to the rank order 
of the pilot district on outcome measures. 



Rank Order Correlation Between: 


Coefficient 


Where n - 6 and p < 


Estimated regional incidence of poverty 
and % of district graduates working 
during the report quarter 


+ .6571 


.10 


Estimated regional incidence of poverty 
and % of district graduates worUng and 
continuing to pursue additional 
education and training 


+ .6571 


.10 


Estimated regional incidence of poverty 
and % of district graduates working 
only (not pursuing additional 
education) 


+ .7143 


.10 


Estimated regional incidence of poverty 
and % of district graduates continuing 
to punue additional education only 
(not working 


- .6571 


.10 



These correlations suggest that economic need may compel graduates to seek work either 
to contribute to household expenses and/or to finance their continued pursuit of 
education and training - fartors beyond the control of education and training providers . 

MSA unemployment rates inversely predia potential labor market demands. The 
greater the demand, the easier it is for recent graduates to find employment. In a '^seller's 
market'' some recent graduates may believe the opportunity costs (e.g., deferred earnings and 
the cost of obtaining services) outweigh the delayed benefits of obtaining additional education 
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and training. Rank order correlations for SMA unemployment rates in ascending order were 
compared to ranked outcome variables. 



Rank Order Correlation Between: 


Coefficient 


'n 

Where n - 6 and p < | 


MSA unemployment rate (descending 
and % of district graduates employed 
during reporting quarter (ascending 


+ .828571 


.03 


MSA unemployment rate and % of 
distria graduates working and 
continuing to pursue additional edu- 
cation and training 


+ .771429 


.05 


MSA unemployment rate and % of 
district graduates working only 


+ .942857 


.01 


MSA unemployment rate and % of 
distria graduates continuing to pursue 
additional education and training only 
(not working 


- .771429 


.05 

■ 



These correlations suggest that regional differences in the availability of work may shape 
the perceived comparative advantages of immediate earnings over anticipated delayed 
benefits (less opportunity costs) and may explain variance in entered employment rates 
of recent high school graduates from the pilot districts. To confirm these hypotheses, results 
from more districts should be analyzed at the regional economic level. Follow-up data files also 
should be linked to in-house data (if available) on former student/family economic status or 
census tract economic data. (Note, too, that rank order correlations of economic variables to 
the percentage of graduates enrolled in publicly fiinded Texas institutions of higher education 
lie in the anticipated direction but were not statistically significant. One would hypothesize 
that stronger correlates will be discovered as the follow-up system expands coverage to 
document enrollments in private institutions of higher education.) 

Among the eight pilot districts, the probabilities for graduates working only (not 
pursuing additional education and training) were highest in the two communities where the 
local economy was dominated by a single major employer engaged in production (rather than 
service) activities -• in particular, labor-intensive firms producing natural-resource based goods. 
Such firms traditionally have employed persons with high school diplomas at comparatively 
attractive wages relative to the skills required. Food and Kindred Products (SIC 20) - chiefly 
grain processing - in Hereford and Lumber and Paper Products (SIC 24) in and around Silsbee 
may account for the greater likelihood that their recent graduates would choftse immediate 
employment over continued pursuit of education and training.* 
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Economists would greet that news with mixed feelings. The long term secular trend for 
the national and state’s economy has been transition from a manufacturing/ processing base to 
service industries. While pockets with high concentrations of manufacturing/ processing remain 
somewhat insulated from those trends, results from six of the eight pilot districts show that 
the top five industries employing recent high school graduates are concentrated in the service 
industries. Among all eight pilot districts, job placements in Eating and Drinking 
Establishments and Grocery Stores were the most likely labor market outcomes followed 
closely by jobs in other kinds of retail businesses. 

Communities where manufacturing/processing jobs are open to recent high school 
graduates are exceptions to the long term secular trend. Proximity to natural resource based 
industries gives them a competitive advantage and insulation from long term secular trends. 
Despite forecasted state and national declines in employment in some of these industries, 
classical economic theory suggests that such opportunities will occur in isolated pockets so 
long as the natural resource(s) are available and demand for the product persists. The capacity 
of manufacturing and processing industries to absorb recent high school graduates in the 
communities where they are concentrated probably will either remain constant or grow. In 
the absence of additional education and training, future high school graduates in the more 
typical communities will face labor market prospects that more closely resemble outcomes in 
the other six pilot districts: i.e., entry-level employment in the service sector. 





County® 
Coefficient 
of Speci- 
alization^ 


Regional* 
Four Year 
Employment 
Growth Rate 


Statewide 
Four Year 
Employment 
Growth Rate 


Forecast 

for Employment 
Growth in Texas 
to Year 2000 


Forecast for 
National Em- 
ployment Growth 
to Year 2005 


Food and 
Kindred 
Products 
(SIC 20) 


11.68 


+ 10% 


- 1% 


- 1% 


- 6% 


Lumber and 
Paper 
Products 
(SIC 24) 


11.58 


+ 52% 


+ 8% 


no growth 


- 3% 



^ For SIC 24, county coefficient of specialization figure for Hardin County which contains Silsbee; for SIC 20, county coefficient of 
specialization for Deaf Smith County which contains Hereford. 

For SIC 20, regional growth figures for Panhandle QWFP Region which includes Hereford; SIC 24 regional growth figures are for Southeast 
Texas QWFP region which includes Silsbee. 



Entry-level employment in service industries (with the exception of Health Semces) 
tends to be low pay with few opportunities for career advancement or earnings gains in the 
absence of additional education and training. Moreover, because service industries are relatively 
"far downstream'^ from goods producing activities, they have smaller "multiplier effects” That 
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is, job placements in the service industries do not add as much value to the regional economy 
as manufacturing/processing jobs because they are less likely to create auxiliary demand for 
"spin off' jobs. And because they are largely "consumed" within a region, services are less likely 
to generate a net flow of income into the region. 

The Texas State Comptroller’s Input/Output Model, based on analysis of sales tax 
revenues generated through business aaivities across industrial seaors, demonstrates the 
relative capacity of job placements to stimulate additional economic aaivity within the state. 
Estimates are not intended to be precise; rather they provide a sense of the order of magnitude 
for ripple effects across industries. While an education and training provider adds value to 
individual graduates by imparting the skills necessary to secure employment in an Eating and 
Drinking Establishment, the relative value added to the economy may only be one-fourth that 
expeaed of the ripple effeas that would result from training and placing graduates in higher 
wage jobs such as with Plastic Produas Manufaauring firms. 

It also is important to note multiplier effeas ripple across the economy from 
placements in newly created jobs rather than from replacement of incumbent workers. That 
is, replacements have no ripple effeas other than forestalling potential loss of economic 
aaivity had positions not been filled. If the growth-to-replacement ratio is low in entry level 
service occupations, the value added to the economy for related placements would be 
considerably lower than those depiaed in the chart below. 



For every ten job 
placements in: 


Estimated Multipliers (Ripple Effects) 


Spin-Off Jobs Added to 
State Economy 


Additional Payrolls 
Generated Statewide 


Additional Economic 
Activity Statewide 


Eating/Drinking Estab. 


3.2 


$ 73,600 


$ 358,000 


Grocery Stores 


5.4 


$ 125,000 


$ 595,000 


Electronics Manufact. 


10.0 


$ 240,000 


$ 1,146,000 


Hospitals 


12.6 


$ 290,000 


$ 1,344,000 


Plastic Manufact. 


12.7 


$ 310,000 


$ 1,524,000 



In Depth Analysis 

More detailed analysis of successful outcomes can be done based on variables in the seed 
records. Data from McAllen ISD are used in the reports below to illustrate the kinds of 
analyses local program administrators may want to condua. By engaging in these detailed 
analyses, they will be able to bater determine where problems may have occurred. In 
particular, they will want to determine how successful they were in tailoring delivery of the 
curriculum to the different needs and learning styles of special populations in their student 
mix. 
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when successful outcomes among McAllen ISD graduates are dis^gregated, there 
appears to be more variance in results between White and Hispanic students than between 
females and males within the same ethnic group. Hispanics particularly Hispanic males) 
appear more likely than other subgroups to forego additional education and training and to 
seek and obtain full time employment upon graduation. Conversely, Whites particularly 
White females) appear more likely to enroll in postsecondary institutions. If they obtain 
employment after graduation, Whites seem more likely to do so as a means of financing their 
continued pursuit of education and as indicated by the higher percentages employed and 
enrolled in school simultaneously.* 

Very little variance is seen in the choice of institution type among McAllen graduates 
enrolling in higher education. The vast majority elected to attend the University of Texas - 
Pan American campus in Edinburgh. The proximity of that university to McAllen appeared 
to be more important than either ethnicity or gender. (In fact, proximity, rather than typ® of 
institution, appears to be the best explanator of where graduates from all eight distrias 
enroll.)Ethnic-based economic differences in continued pursuit of education and training inight 
emerge, however, if the follow-up system gains access to enrollment information from private 
institutions whose tuition costs tend to be greater. 

Seventeen students in the cohort graduated from McAllen ISD after finishing Individual 
Employability Plans. The number of cases was too small for detailed analysis. Among regular 
graduates, graduates of advanced programs, and graduates of advsuxced programs with honors, 
there were notable differences in outcomes. As expected, advanced program graduates and 
those with honor were more likely than regular graduates to continue to higher education; m 
particular, of those enrolled in institutions of higher education, both subgroups were more 
likely than regular graduates to attend a university. Perhaps because of the availability of 
merit-based scholarships, those with honors were by far the least likely to forego continued 
education for full time work or to work while going to school.’ 

Among special populations^*^, too few special education and ESL students graduated 
from McAllen ISD for detailed analysis. However, of all subgroups analyzed, special education 
students were the least likely to either find employment or continue their pursuit of education 
and training. Except for the special education graduates, members of special populations were 
more likely than their classmates to forego additional education and training to obtain work, 
particularly at full-time jobs. Limited English proficiency, given the proximity of the district 
to Mexico, does not appear to be a barrier to employment among McAllen graduates. 

Special populations categories are not mutually exclusive; students can be classified as 
falling into more than one of the categories. Given a sufficient number of cases, local program 
administrators are advised to compare outcomes for those in multiple categories to outcomes 
for those meeting only one special populations criteria. Local program administrators also 
should test for the auto-correlation between lEP graduates and various special populations wte- 
gories (special education in particular) before attributing significance to presumed relationships. 
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In analyzing public education outcomes, researchers should be mindful of data 
limitations: 

1) Limitations of records selected for inclusion in the seed files. 

a) Inadequate means of distinguishing vocational students from others in the 1992-93 cohort. 

Since the follow-up of public education students was funded with Perkins dollars, the 
primary intent of the study was to document successful outcomes for 1992-93 graduates who 
had received vocational training. Records were selected where the value in the PEIMS data 
base for the vocational education variable equalled "1". Unfortunately, the coding standards 
for submission of data to PEIMS for the 1992-93 cohort failed to distinguish completers of 
coherent sequences of vocational courses from the "incidental”^^ vocation^ course-takers. The 
code "1" was assigned to any student who during the reporting "window" in October was en- 
rolled in at least one vocational course. (The code "1" does not even provide assurance that the 
student completed a vocational course.) In 1993, PEIMS added a separate code for coherent 
sequence completers. Not until 1994 was another code added to distinguish Tech Prep 
students. Comparisons of outcomes among regular graduates, vocational graduates, and Tech 
Prep graduates can not be done on the 1992-93 cohort. 

RECOMMENDED ACTION 

PEIMS has.been asked to correct the extraction routine to add seed records for students 
not enrolled in a vocational course during the October reporting window for the 1992- 
93 cohort. In future studies, coherent sequence completers and Tech Prep graduates will 
be differentiated as new values were added to the vocational education variable in 1993 
and 1994 PEIMS standards respectively. 

b) Selection of graduates only. 

Funding limitations forced selection of graduates only. Therefore the analysis can not 
address important research questions related to the comparative results for graduates and drop- 
outs. 



RECOMMENDED ACTION 

PEIMS has been asked to correct the extraction routine to add records for members of 
each senior class who did not graduate with their cohort and who were not still 
enrolled in secondary education. 
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c) Invalid Social Security numbers (SSNs). 

Pilot distrias were seleaed because they reported available SSNs to PEIMS. Across the 
state, SSNs are reported for 87.9% of all public education students (ranging from a high of 
100% to a low of 43.8%. Among the eight pilot districts the availability of SSNs ranged from 
72.8% to 97.6%.) However, not all SSNs reported to PEIMS are valid nor are all those 
contained in the administrative data bases linked by record matching techniques. In Program 
Year 1994-95, the follow-up system will use an edit check routine (derived from SSN 
assignment algorithms) to capture invalid numbers in seed records. During the 993-94 study, 
in the absence of automated edit checks, records were discarded as having an invalid SSN only 
if manual inspeaion revealed a mismatch between the name entered in the student record and 
the name entered in the matched UI w^e-record.“ 

Biases in missing or invalid SSNs may skew results. Where immigrants are concen- 
trated along the US-Mexico border or in enclaves within the state’s urban areas, a dispro- 
portionate number of students may not have SSNs. Moreover, it has been suggested by 
praaitioners that sharing of SSNs and/or their fraudulent use may be concentrated among 
immigrants and in economically depressed areas. 

RECOMMENDED ACTION 

Districts reporting SSNs for fewer than 87.9% (the state average) of their graduates 
should be encourage to be more aggressive colleaing that information. Distria coor- 
dinators are encourage to use the edit check software derived from Social Security 
Administration algorithms as a means of assuring quality control prior to submission 
of data to PEIMS. This will allow distrias to validate and correa SSNs rather than 
having records discarded as unusable by the follow-up system. Local program adminis- 
trators are encouraged to use conventional mail and telephone survey techniques to 
condua supplemental follow-up on students for whom vdid SSNs were not available. 
Results of these supplemental follow-up should be shared with the SOICC. Only by 
comparing results for those with valid numbers to the results documented through 
supplemental follow-up can the system determine response-set biases and devise 
appropriate statistical correaions in performance calculations. 

2) Considering other explanatory variables. 

Note in the public education seed record file layout (Appendix II) that several fields 
were left blank. Upon release of preliminary data analysis to the pilot distrias, it was 
determined by a consensus of site liaison that variables available in PEIMS should be added 
in the requested rerun: Had the graduate been enrolled in a gifted and talented program? Had 
the graduate expressed an intent to pursue additional training and education after high school? 
Had the graduate participated in a Work-Study program? Had the graduate participated in a 
Cooperative Vocational Education program? 
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Cooperative Vocational Education program? 

Two others variables were considered for inclusion but were not added. While the 1992- 
93 PEIMS standards included an indicator for student participation in programs to eliminate 
sex-biases in occupational training, that variable is not available for subsequent cohorts. The 
liaison elected not to include that indicator because it did not lend itself to comparisons over 
time to see if such programs were improved. An "at-risk” indicator is available for the 1992-93 
and subsequent cohorts. After discussion, it, too, was not requested in the rerun because 
controversy surrounds the varied practices of ISDs in coding their students at-risk. Moreover, 
proxies for at-risk status could be deduced from more clearly defined and consistently coded 
special populations variables. 



RECOMMENDED ACTION 

Local program administrators are not consider limited to the explanatory variables 
selected by the System Director and site-based public education liaison for inclusion in 
seed records. After matching, files are returned to the districts with SSNs neither 
stripped nor encrypted. This allows the files to be relinked to in-house data bases. As 
locd administrators ask questions not answered by standard follow-up independent vari- 
ables, they should exercise the option of appending additional explanatory variables to 
the end of each record and engaging in their own analyses. Results of supplemental 
analyses should be shared with the SOICC. Where in-house variables explain signi- 
ficant portions of variance in outcomes, recommendations should go forward for their 
inclusion in all seed records. Where a powerful explanatory variable is unique to a 
particular district, its statewide adoption in PEIMS should be moved for consideration 
by the State Board of Education. 

3) Determining program level outcomes. 

In the absence of an appropriate code value for the vocational education variable, there 
is no way for the system to differentiate coherent sequence (or program) completers from 
incidentai vocational course-takers. Even when the appropriate code value is available for the 
1993-94 graduating cohort, it still will not be possible to disaggregate outcomes by program. 
While coherent sequence completers will be distinguished from incidental vocational course 
takers, there will be no way to use PEIMS data, for example, to sort automotive program 
completers from those in a coherent sequence related to electronic technology. 

The difficulty is further compounded by the use of a system unique to Texas for coding 
course enrollments (the TEA code). Unlike the Classification of Instructional Programs (CIP) 
system used by higher education and the JTPA system to code courses, TEA course codes can 
not be translated or aggregated into program level data.‘^ Program level information is crucial 
in determining training-relatedness of job placements. It is the completion of an entire 
coherent sequence that prepares a student for occupational employment; it would be 
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inappropriate to attribute employability to completion of a single course. In the absence of 
a method for comparing job placements to program completion, the occupational titles held 
by former students (as collected in the employer follow-up survey) are interesting but not 
susceptible to meaningful analysis. We will be limited, in all practicality, to addressing the 
question: "Does the completion of any coherent sequence of courses improve the likelihood 
of successful outcomes?" Comparisons of outcomes among coherent sequences - even of like 
title - would be relatively meaningless. 

Four possible solutions have been suggested; 

a) Manual identification of program completers. 

Until the appropriate code values are available in PEIMS data on subsequent graduating 
cohorts, local administrators may either link returned files to their in-house data bases 
or ask vocational directors and instructors to manually identify program completers 
from class records. In most districts this process probably can not be automated 
immediately and would create inefficiency in the system, uncoordinated and potentially 
non-standardized analysis, and added burdens on local personnel. 

b) Convert from TEA to CIP codes. 

Con^'ersion of public education from the TEA <x>ding system to the CIP system would 
require a massive effort to recode all courses and to update all related report forms, the 
PEIMS data base struaure, and related materials both at the state and local levels plus 
an exhaustive technical assistance effort to explain the changes. This would be expensive 
and time<onsuming. 

c) Develop and continuously update a TEA-to-CIP crosswalk. 

The TEA is required to report the use of federal education dollars by CIP code. Work 
is already done at the agency level to translate TEA codes into their CIP equivalents. 
While ^ency personnel constantly validate and improve that crosswalk, technical 
assistance should be provided to help local administrators/researchers understand and 
use the CIP coding system to make sense of analyses of training-relatedness for job 
placements. A program would have to written to scan the CIP codes assigned to 
courses taken by public education students, determine at the six digit level which CIP 
program best fit the individual student’s pattern, and assign that as the program level 
variable in PEIMS. Programming would have to be coupled with a substantid develop- 
ment and validation effort. 

d) Use of capstone experience course code as a proxy for program level data. 

Every coherent sequence of courses is supposed to end with a capstone experience. Each 
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capstone experience has a TEA course code. Rather than scan across all career and 
technology courses taken by a public education student to determine the best fitting 
CIP program code assignment, allow the capstone experience to stand as a proxy for 
the whole coherent sequence taken. The TEA-to-CIP crosswalk would then be needed 
only to convert the capstone course code into its CIP program level equivalent. From 
there, determinations of training-relatedness could be made without using more 
cumbersome file scanning software. 

RECOMMENDED ACTION 

Use of the capstone/ CIP variable would be the most efficient but might be subject to 
misinterpretation. Great care would have to be taken in the narrative portion of 
outcome reports to note that the capstone experience was a proxy for the program 
consisting of the capstone and the preceding courses taken in the coherent sequence^ i.e., 
to avoid attributing employment outcomes to the capstone experience alone. 

Another factor complicates the interpretation of program outcomes in public education. 
That is, each district is allowed to determine (within a narrow range) which combination of 
courses constitutes a coherent sequence for each program area. There is, for example, no 
standardized public education program called "Automotive Technology." Therefore, it would 
be inappropriate, even upon analysis of the entire population of Texas high school graduates 
for a single cohort, to assert that, statewide, any particular program was successful. Again, care 
must be taken in descriptive narrative accompanying outcomes analyses. 

The most one could say, for example, at the district level would be: "Among those com- 
pleting the coherent sequence of counes this district calls Automotive Technology*, those enrolled 
in the program at Campus X were more likely to find related work after graduation than were 
those enrolled at Campus Y;" or "Among Hispanic males, higher success rates were achieved by 
those who completed the district*s Automotive Technology’ program than by those who completed 
this district’s 'Construction Trades’ program. " 

In such cases, administrators would want to know if the Automotive Technology 
programs at Campuses X and Y served different student mixes. If not, why was the program 
on Campus Y, less effective? Were there significant differences in regional occupational 
demands in the auto repair and construction industries? If not, why was the Automotive 
Technology program more effective in delivering services to the Hispanic male subgroup? 

In making cross-district comparisons, inferences would have to be limited to such 
statements as: "Completers of what District X called Automotive Technology’ were more likely 
to obtain full time employment after graduation than were completers of what District Y called 
Automotive Technology. ’" Perhaps District Y should consider realigning its Automotive Tech- 
nology curriculum to match the coherent sequence adopted under that title by District X. 
(Perhaps after comparison of results by neighboring distrias, local variations in program 
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definitions will disappear - not by imposition of statewide uniform definitions, but because 
best practices will be identified and emulated. 

Patterns of Occupational Employment Among R.ecent High School Oraduates 

Patterns of occupational employment among high school graduates from pilot districts’ 
1991-92 cohort are offered as "interesting" - but not "conclusive" - findings. Again, one must 
remember that the districts were selected from a pool of volunteers and were not repre- 
sentative of the state’s public education system. Nonetheless, results were predictable. 

While 75% of the public education completers found work after graduation, 32% did 
so on a full-time basis. Of the entire graduating cohort for all eight pilot districts, less than 
one-fourth secured full time employment. Of those who worked full time for the full repor- 
ting quarter, average quarterly earnings were $ 2,277 (only 3.3% higher than minimum wage). 
Employment was most likely to be found in the service sectors; 53% were employed in five 
service industries: Eating and Drinking Establishments (21%), Food Stores (11%), General 
Merchandise Stores (9%), Business Services (7%), and Apparel/ Accessories (5%). The 
occupations most commonly open to 1991-92 pilot district graduates (both part time and full 
time) were: Retail Salesperson, Cashiers, Food Preparation Workers, Waiters/ Waitresses, and 
Helpers/Laborers. 

The best use of these data would be to use them in career counseling. By comparing 
public education and higher education outcomes by entered employment rates, percentage of 
subjects obtaining full-time/full quarter employment, and average quarterly earnings for full- 
time/full quarter employment, the data demonstrate that employers prefer to individuals with 
higher competencies than can be attained and demonstrated by high school graduates. While 
there are exceptions, higher paying jobs are available to persons with some college. As 
educational attainment levels increase, so do potential earnings. 

Earnings Levels for Samples of 1991-92 Graduates in the Employer Follow-Up Survey 
for Employees Identified as Full Time/Full Quarter 





Annual 


Hourly 




Annual 


Hourly 


High School Pilots 


$ 9,096 


$4.37 


JTPA Title IIA 


$11,428 


$ 5.49 


Com/Tech College Certificate 


$16,140 


$7.76 


JTPA Title in 


$17,428 


$8.38 


Com/Tech Associate Degree 


$23,312 


$11.21 









This is not a criticism of the public education system. Rather, it reflects the realities 
of the labor market where Texas firms must compete globally. Public education has no control 
over employer demands and expectations. Occupational employment outcomes for recent grad- 
uates, therefore, are better used for motivating students to continue their pursuit of education 
and training after high school. In evaluating the performance of public education, it may be 
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wise to consider giving more weight to indicators of graduates’ abilities to gain admission to 
and successfully complete more advanced education and training. To that end, public 
education practitioners have suggested that future follow-up studies include matching seed 
records to such things as TAAS, ACT and SAT scores as indicators of their preparedness for 
higher education. Such outcomes information might be more meaningful to public education 
because it relates more directly to the factors over which practitioners have some degree of 
control. 

It is the SOICC’s long range plan to include follow-up data ^^regated to the program 
level to add "track record^ information and "likely outcomes" to the SOCRATES Q>lanning) and 
Texas CARES (automated career information delivery system) programs to guide and motivate 
prospeaive students and adult learners in their career decision-making. 



Results for the JTPA System 

The primary reason for conducting a pilot study for the JTPA system in Texas was to 
obtain follow-up data through the automated record linkage technique and compare results to 
those obtained through traditional telephone survey techniques by Texas A&M University 
under contract to TDoC. Do both methods yield similar findings? Are the differences, if any, 
significant? What would be the benefits of replacing the traditional telephone survey with 
automated follow-up? What, if any, would be the disadvantages? 

Comparisons of data collected by the two methods on the same cohort of JTPA leavers 
reveals that nearly identical results are obtained. Automated follow-up found 369 more former 
participants (14%) employed in the matched quarter than did the telephone survey conducted 
by Texas A&M University at the thirteenth week after program termination. Automated 
follow-up also identified 289 persons (11%) who were not included in the TEC UI wage- 
records in the report quarter but who, at the thirteenth week, told Texas A&M interviewers 
that they were employed. (In sum, the record matching technique provided the JTPA system 
with a net 3% gain in the number of documented successful outcomes.) For more than 75% 
of the JTPA Title IIA participant cohorts studied, automated follow-up and the telephone 
survey obtained the same results. Any differences in findings between the two methods, 
therefore, are insignificant. 

Data collection timeframes and definitions of employment can account for most 
differences. For those found in the UI wage records but who reported themselves unemployed, 
differences may be explained by the timeframes covered in the data. Both results could be true 
for participants whose employment status changed and for whom the 13th week Texas A&M 
survey did not fall in the 4th quarter of 1993. For those who reported themselves employed 
but who were not located in the UI wage records, there are two possibilities. First, as above, 
employment statuses may have changed between the two data collection points. The other 
possibility: they were employed during both timeframes but the jobs held did not meet the 
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employment criteria for inclusion in the UI wage records, i.e., employment in a job covered 
by the state’s Unemployment Compensation Act. 





4th Quarter 1993 UI wage-records 


Not employed in a job 
covered by Texas State 
Unemployment Com- 
pensation Act 


Employed in a job 
covered by the Texas 
State Unemployment 
Compensation Act 


Texas A&M University 
Telephone Survey of 
Former Participants at 
13th week after 
termination 


Self-Reported Not 
Employed 


607 


369 


Self-reported Employed 


289 


1,422 



T (test statistic)- 13.8; cht-square with one degree ot treedom at p < .001 is 10.8} 



Given the comparability of findings, we conclude that automated follow-up results are 
as reliable as those obtained by the telephone survey technique. Additional questions must be 
addressed before a policy decision can be made regarding a request to the Department of Labor 
(DOL) for a waiver to substitute automated follow-up for the traditional survey.*^ The chart 
be!aw outlines the basic considerations. 



r 


Automated Follow-Up 


13th Week Telephone Survey 


II Data Source(s) 


Administrative records + employer survey 


Self-reported/U nverif ied 


I Cost 


Less than $.50 for automated matches; 
approx. $5.00/record for inclusion in 
employer survey 


Estimated at $13.26/record 


Timeframe 


Single point in time: 4th quarter 


Continuous: 13 wks after exit 


Availability 


Five months after close of 4th quarter 


Quarterly/year-end repons 


Employment Outcomes 


Yes 


Yes 


Earnings Information 


Quarterly earnings and FT/FQ flag 


Hourly wage (unverified) 


Participant Satisfaction 


No 


Yes 


Continued Education 


Match to Coordinating Board Master Files 


No 


Capacity to identify 
long term outcomes 


So long as participant remains in covered 
employment in Texas, military, fed civil 
service, postal service, Texas higher 
education, or surrounding state with data 
sharing agreement, any longitudinal wave 
will obtain data 


Significant decline in the 
number of panicipants 
reached by telephone as 
accuracy of contact 
information in the 
panicipants’ files decreases 
over time. 
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The second major purpose for inclusion of JTPA programs in the 1993-94 pilot study 
was to identify barriers to coordinated program planning and evaluation. Conforming amend- 
ments in the JTPA and Perkins Acts, the Texas WorMorce and Economic Competitiveness 
Act, and proposed reforms all call for increased integration and coordination of workforce 
development programs — including integrated planning and evaluation. To the extent that data 
definitions and performance calculations were established independently by the various state 
agencies prior to the push, existing practice may impede coordination. 

One JTPA program evaluation practice in particular was inconsistent with those used 
by other workforce development agencies participating in the 1993-94 follow-up study. Like 
public education and higher education, the JTPA system conceptually defines training-related 
placements as successful outcomes; in Texas, it differs in its operational definition of training- 
relatedness. Whereas other service providers assign a training program codes to participant 
interventions and occupational codes to employment outcomes, the JTPA system assigned an 
occupational code to outcomes and interventions . That is, the JTPA system records the OES- 
of-training as the code representing services received and the OES-of-placement for employ- 
ment outcomes. Higher education, on the other hand, assigns a CIP code to the training 
provided and an OES code to employment outcomes. 

Coding practices in this case make a significant difference in the way program 
performance is calculated. To receive a positive score for training-related placements, there had 
to be a one-to-one correspondence between the JTPA participant’s OES-of-training and OES- 
of-placement. Placements in similar (but not identical) jobs for which they were trained were 
not counted as successful outcomes. For each CIP code, however, there may be more than one 
related occupational title. Using the CIP system to code training services provided, therefore, 
increases the likelihood that job placements will be considered training related. Other 
workforce development partners using the CIP coding system will appear to have better 
performance ratings on the training-relatedness measure. That would put the JTPA system at 
a decided disadvantage when cross-program comparisons are made in an integrated evaluation 
system. 



A simple example illustrates the problem. An SD A purchases automotive training off- 
the-shelf from a local community college on behalf of a JTPA participant. Upon program 
completion, the participant secures employment as a diesel mechanic. The participant is 
included in both the JTPA and community college follow-up. 



Entity 


Training Code 


Outcome Code 


Training-Related? 


JTPA SDA 


OES « 85302 


OES - 85311 


No 


Community College 


CIP - 740603 


OES - 85311 


Direct Relationship 



Same individual/same training/same outcome would be rated differently for no other reason 
than the differences in data coding practices. 
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Consistency for the sake of cross-program comparisons in an integrated program 
evaluation system could be achieved either by asking the JTPA system to code training services 
according to the CIP system or by asking the community colleges to use OES-of-training 
codes. Conversion of the JTPA system to CIP codes better refleas both the way the 
curriculum is designed and the struaure of employment opportunities. Most programs 
(outside those for licensed and regulated occupations) impart a broad enough range of 
compaencies to prepare graduates for a cluster of related jobs. Employers, in turn, hire 
applicants whose combination of compaencies provides a satisfactory fit with job requirements 
and whose aptitudes and interest suggest that they can quickly acquire all other related 
compaencies. Requiring a one-to-one correspondence baween OES-of-training and OES-of- 
placement nndlerestimates the portability of compaencies across a cluster of related occupations 
and overestimates the rigidity of employers’ applicant-screening and job-assignment praaices. 

An ad hoc group consisting of the Follow-Up System Direaor, the employer survey 
subcontraaor and SDA/follow-up liaison (led by Mary Ross of the West Central Texas 
Council of Governments) brought this matter to the attention of Jim Boyd, Direaor of 
Workforce Development Division, TDoC. Mr. Boyd and his staff reviewed the data, con- 
sidered the illustration provided, and compared performance calculations. They concurred with 
the recommendation to convert the TDoC’s management information system to the use of 
CIP codes for instruaional services rendered to JTPA participants. 

Mr. Boyd also daermined that conversion to CIP codes would be praaical insofar as 
providers of off-the-shelf training programs purchased on behalf of JTPA participants already 
used the CIP coding system and could supply that information when contraaing with the 
SDAs. Moreover, the DOL guidelines permitted (and its chief administrators advocated) 
conversion to the CIP coding system. In a Standardized Program Information Report 
memorandum to state JTPA direaors, the DOL announced: 

The CIP was chosen as the preferred coding system for classroom training programs because 
it is the only classification system... specifically desired to describe at^ code educational 
programs. Even "occupationalfyspedfic" educational programs tend to prepare for a job 
family or cluster of occupations with related competencies rather than for a single 
occupation. Generally, it is not advisable to try to assist a single occupational code to an 
educatiorml training program.'^'' 

JTPA SDA direaors and MIS coordinators were notified of TDoC’s intent to convert 
to CIP codes. Programming changes were made in the MIS. SDA administrators wer«. notified 
of the change and arrangements for technical assistance were made before the close of the 
program year per the memorandum reprinted on the following page. The technical assistance 
rendered to TTDoC and the agency’s prompt response will increase SDA acceptance of the 
follow-up system and the likelihood that outcomes information will be used properly and 
fairly in an integrated program evaluation process. 
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STATE OF TEXAS 

DEPARTMENT OF COMMERCE 

MEMORANDUM 



TO: 


SDA Administrative Entities 


FROM: 


Joan Kotal-ift — ^ 


DATE: 


i) 

June 21, 1994 


SUBJECT: 


CMS: NEW CIP/FICE CODE INFORMATION 



The JTPA Data Analysis Committee chaired by Mary Ross of West Central Texas 
COG, in conjunction with the SOICC Student Follow-up Project, has brought to the 
forefront the importance of comparing the training-relatedness of JTPA placements 
to other partners in work force education and training programs. The committee 
concluded that a comparison of the JTPA OES activity/training/ placement codes to 
the educational/training codes does not allow for an adequate assessment of the 
relationship of placements to the training activity. The use of the Classification of 
Instructional Programs (CIP) code system to allow multiple training programs to 
correspond to an Occupational Employment Statistic JOES) job placement is necessary 
for accurate evaluation. The educational institution/training provider is assigned a 
FICE code, which will be collected to facilitate future evaluation of institutional 
effectiveness. 

The Client Management System (CMS) will be modified in the conversion process to 
include both the CIP and the FICE code in the CMS Software Participation Module and 
the Participant Coding Sheet effective PY94. The CIP code will be used In 
conjunction with the OES code to classify JTPA training activities, excluding On-the- 
Job training participants which only requires an OES code. The OES code will 
continue to be collected for placement. Because many participant coding sheets are 
completed by the educational institution/training provider, it should not be difficult to 
obtain the CIP and FICE codes by the local JTPA program operators. 

A list of CIP codes and examples of how the CIP code relates to the OES code are 
included for review. The CMS system will have the CIP data available as a choice 
screen in the Participation Module, but the FICE code should be provided by the 
educational institution. If you have questions, or need further information, please 
contact Teresa Alvarez at (512) 320-9813. 

JK:ta 




Post Office Box 1 2728 • Austin, Texas 7871 1-2728 • 512/472-5059 
TDD: 512/320-9698 * Relay Texas Une: 800/735-2988 

htntad lUofM 
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Results for JTPA varied significantly across titles. The variance was not unexpected 
given the different goals, objectives and eligibility criteria for each program. 



JTPA PARTICIPANT OUTCOMES 
Program Year 1992-93 



10-mi 


Number 


Enrolled in 
Higher Ed 
Fall Term 


Emfdoyed in 
4th Quar- 
ter 1993 


Emidoyed 8c 
Enrolled in 
Higher Ed 


Either Em- | 
ployed or | 
Enrolled g 


Tide IIB: Summer 
Youdi Programs 


50,815 


8% 


29% 


3% 


33% 1 


Education Coordin- 
ation Plans 


9,161 


5% 


30% 


1% 


30% 1 


Older Worker 
Program 


2,275 


1% 


41% 


0% 


42% 1 


Tide IIA: Economically 
Disadvantaged Youth 


22,430 


7% 


42% 


4% 


45% 1 


Tide IIA: Economically 
Disadvantaged Adults 


15,795 


6% 


64% 


4% 


66% 1 


Tide ni: Dislocated 
Worker Program 


12,912 


7% 


72% 


5% 


74% I 



The Title m program is designed for rapid deployment of resources to help dislocated 
workers reenter the labor market as soon as possible after plant closures and laige scale layoffs. 
Title in participants already have work histories and are more likely than those in other JTPA 
programs to have relatively high levels of education and competencies acquired through experi- 
ence. As expected, that program had the highest rate of successful labor market outcomes: 72% 
found work and annualized earnings were more than 50% higher than for Title IIA program 
completers who found full time work. At the other end of the spectrum are various JTPA 
youth programs. These are designed largely to keep at-risk youth in high school and to 
provide dropouts with the GED. Enrollments in higher education are low because youth 
program completers may not yet be of college age by the start of the subsequent fall term. 
Additional automated matches are recommended to give a better picture of youth program 
successes: 

• JTPA IIB youth program seed records should be matched against TEA PEIMS data to 
determine what percentage completed their high school education within one, three and 
five years after receiving services. 

• JTPA IIB and IIA youth program seed records should be matched against adminis- 
trative records to determine what percentage earned a GED within one, three, and five 
years after receiving services. 
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• JTPA nB and HA youth program seed records should be matched gainst higher 
education enrollment records again at the three and five year intervals to determine 
what percent^e subsequently sought additional training and education. 

• Pre-enrollment/post-exit outcomes should be compared to determine the earning gains 
made by Title IIA (Adult) and Title HI program completers. 

JTPA participants are often eligible for multiple services. In Title IIA Adult programs, 
for example, some participants eng^e in their own job search activities independent of JTPA 
provided services. Under family or financial pressure to enter the labor market, they may take 
jobs before completing JTPA-provided occupational skills training.. However, follow-up data 
indicate that those who use the full compliment of services were most likely to have successful 
labor market outcomes. 



JTPA Interventions 


Received Job Placement Assistance 


Mix of Services 




No 


Yes 


Completed Occupational 
Skills Training Program 




4,020 


3,645 




No 


42% 


70% 


(Each cell shows Number 
of participants, % em- 




$1,944 


$2,653 


ployed, average quarterly 
earnings 




1,751 


6,379 




Yes 


52% 


78% 






$2,467 


$3,517 



SDA administrators are encouraged to conduct comparable analysis for their other 
multiple intervention services to determine which combinations produce the highest success 
rates. In particular, they should examine the impacts of needs-based payments and varieties of 
special support services. 

Because, on averse, participants in Title IIA and Title III programs were older than 
subjects in the public education pilots, many already had a high school diploma or GED. Age 
and experience may explain why former JTPA participants were placed in a different mix of 
industries and occupations than were graduates from the public education pilot districts. 
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JTPA TITLE DA 


JTPA TITLE in 


Top 5 Industries by 
placements (per UI 
wj^e record match) 


Top 5 Occupations by 
placements (per 
Employer Survey) 


Top 5 Industries by 
placements (per UI 
wage record match) 


Top 5 Occupations by 
placements (per 
Employer Survey) 


Health Services 14% 


Helpers/Laborers 


Business Services 13% 


General Office Clerks 


Business Services 13% 


Cashiers 


Health Services 7% 


Secretaries 


Eat/Drink Estab. 11% 


Nurses Aides 


wholesale Trades 6% 


Truck Drivers 


Food Stores 5% 


General Office Clerks 


Educational Services 6% 


Retail Sales Persons 


Educational Services 4% 


Retail Sales Persons 


Government 5% 


Helpers/Laborers 



SDA administrators should use these outcomes data to validate the logic used in 
planning services. Appendix IV gives more detailed instructions. 



Results for Higher Education 
Major Findings 

For the third consecutive year, automated follow-up has documented successful 
outcomes for 85% or more of the community and technical college cohorts studied. Of the 
students in the 1992-93 School Year followed under this grant, 87% were either employed 
and/ or continuing to pursue additional education and training. 

In the table on the next page, results are broken out by graduation type. All graduates 
of community and technical college in the state were followed. They included academic, 
technical and tech prep graduates. All academic and technical non-returners also were followed. 
Six community colleges participated in the adult vocational follow-up. They were: 

Amarillo College Brazosport College Del Mar College 

El Paso Community College District McLennan Community College 

North Harris/Montgomery Community College District 

Six Universities participated in the 1993-94 pilot. They were: 

East Texas State University Lamar University 

University of Houston - Clear Lake The University of Texas -Dallas 
The University of Texas - Pan American West Texas A&M University 
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1 1992-1993 

1 School Year 

1 Cohort 


Pursuing 
Additional 
Ed. Only 


Employed 


Both; 

Working + 
Add. Ed. 


Total*: 1 

Employed or || 
in School 


Adult Vocational - 6 pilot colleges 


3% 


69% 


8% 


79% 


(N - 19,110) 


504 


13,159 


1,474 


15,137 


All Com/Tech College Graduates 


7% 


80% 


87% 


(N - 32,086) 


2,246 


25,669 


228,129 


Academic Graduates 


16% 


29% 


43% 


88% 


(N - 8,221) 


1,294 


2,412 


3,560 


7,266 


Academic Non-Returners 


8% 


52% 


25% 


86 % 


(N - 242,052) 


19,786 


127,052 


61,568 


208,406 


Technical Graduates 


4% 


68% 


14% 


86% 


(N - 23,721) 


989 


16,089 


3,325 


20,403 


Technical Non-Returners 


4% 


68% 


13% 


85% 


(N - 138,579) 


5,231 


93,843 


18,075 


117,149 


Tech Prep Gradiutes 


4% 


73% 


10% 


87% 


(N - 144) 


6 


105 


14 


125 


Universities - 6 pilots 


3% 


73% 


12% 


88% 


(N - 8,561) 


229 


6,268 


1,062 


7,559 



Totals subject to rounding errors. Source: Texas Higher Education Coordinating Board 



Of those pursuing additional education, type of transfer institution varied by category 
of community and technical college major pursued. 



Of Students Pur- 
suing Additional 
Education: 


Returned to Same 
Institution for 
Additional Education 


Transferred to 
Another Community 
or Technical College 


Transferred to a 
University or Health 
Science Center 


%Full 

Time 


Academic Grads 
(N - 4,854) 


19% 


4% 


77% 


59% 


Academic Non- 
Returners 
(N - 19,786) 


N.A. 


21% 


79% 


72% 


Technical Grads 
(N - 4,314) 


65% 


11% 


30% 


70% 


Technical Non- 
Returners 
(N - 5,231) 


N.A. 


42% 


58% 


54% 


Tech Prep Grads 
(N - 20) 


85% 


5% 


35% 


65% 
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Technical graduates who transferred to universities or health science centers were most 
likely to major in Registered Nursing (501); Undeclared (323); Business Administration and 
Management (216); Respiratoiy Therapy (160); Practical (UPN) Nursing (143); Administrative 
Assistant/Secretarial Science (130); Criminal Justice (126); Accounting (114); General Business 
(111); and Computer Science (110). 

Academic graduates who transferred to universities or health science centers were most 
likely to major in Multidisciplinary Studies (482); Undeclared (361); Business Administration 
and Management (356); Accounting (270); General Business (253); Psychology (235); Registered 
Nursing (214); Biology (144); English (143); and Health and Physical Education (112). 

Of those from the 1991-92 cohort entering the labor market, 58% were employed in 
10 industries in the service secton Health Services, 35,964; Eating and Drinking 
Establishments, 31,742; Educational Services, 28,473; Business Services, 27,033; General 
Merchandise, 12,708; Food Stores, 12,708; Miscellaneous Retail, 11,116; Federal Government, 
10,828; Social Services, 10,143; and Apparel and Accessories, 8,564. According to the 
Employer Survey, the top five jobs held by community and technical college graduates and 
non-returners who entered the labor market full time were: Registered Nurse, Licensed 
Practical Nurse, Respiratory Therapist, General Office Clerk, and Secretary. 

Changes at the Leading Edge of Follow-Up 

Higher education has participated longer in automated follow-up than the other 
workforce development partners. Coordinating Board staff members compute statewide success 
measures and distribute data files to each institution of higher education. Each college or 
university has at least a half-time institutional researcher qualified to use the data in 
compliance reports, self-studies for accreditation, and internal planning for continuous program 
improvement. At the system level, their experience and expertise is no longer focused on the 
mechanics of automated record linkage. Rather, they are be^nning to address more sophis- 
ticated and detailed questions pertaining to the improvement of follow-up itself. 

Improving the Breadth of Coverage 

Higher education practitioners took the lead in the Steering Committee and in informal 
discussions to improve the breadth of coverage for the follow-up system. Whereas in seed 
records only from nine volunteer conununity and technical colleges were included in the 
employer follow-up survey during Program Year 1992-93, the Coordinating Board extracted 
seed records for all graduates (of both academic and technical programs) and a sample of non- 
retumers from all 49 technical and conununity colleges in Program Year 1993-94. Five univer- 
sities joined East Texas State University (from the prior year) in the match against higher 
education master enrollment files and the UI wage records. An attempt was made to devise 
a method for ascribing outcomes to adult vocational programs. 
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Praaitioners from institutions on the perimeter of the state, particularly Dr. Mike Wolf 
of El Paso Community College, were helpful in contacting surrounding states’ officials to 
begin negotiations on data sharing ^reements. Members of the LoneStar Data Users group 
coordinated aaivities with the Follow-Up System to improve the quality and coverage of 
information available at the campus level either for inclusion in the basic follow-up seed 
records or for detailed analysis when returned files are linked by institutional researchers back 
to in-house data bases. In particular. Dr. Stan Adelman was a persistent advocate for the use 
of student intent information on file within the institutions in the analysis of outcomes. 
Members of the Elearonic Transcript Group were influential in developing linkages that will 
allow institutional researchers to examine grades earned at transfer institutions as outcomes; 
i.e., as indicators of the preparation received at the institution exited. Richard Bailey, Dr. 
David Preston and Dr. Mike Green were particularly hdpful in explaining the parallels 
between automated student follow-up and elertronic transcript analysis. Their work convinced 
public education practitioners of the need to examine student intent to enroll in college when 
analyzing outcomes and examining their graduates’ TAAS scores as indicators of college 
preparedness. 

Don Perry, President of TAPSOEA and Steering Committee representative from the 
Dallas County Community College District, was the first to voice concerns about the way 
gaps in UI wage record coverage disproportionately effect programs which prepared students 
for occupations in traditional areas of self-employment: e.g.. Cosmetology, Real Estate, and 
Music. TAPSOEA members were instrumental in the System Director’s efforts to devise 
supplemental follow-up procedures in collaboration with Coordinating Board staff. 

Supplemental follow-up procedures allow institutions/service providers to document 
successful outcomes for former students not located in the link&i records tapped by 
automation. Seed records would be returned to institutions along with instructions or 
software for extracting the "exceptions list" of persons not located. Institutions would have 
the option of conducting follow-up through conventional telephone and mail survey tech- 
niques to locate former students at last known point of contact. Institutions would be given 
a deadline for the return of verifiable information in a standardized format. The format 
would require submission of data items comparable to those obtained for other students via 
record linkages and the employer follow-up survey. Until the deadline, all successful outcome 
rates would be released as "preliminary - subjert to supplemental follow-up." After the 
deadline, verifiable supplement data would be used to recompute all institutions' "official" 
successful outcomes ratings. 

Coordinating Board staff, led by Helen Giraitis and Dr. David England, were persuasive 
in expansion of the conceptual definition of successful outcomes to include transfers to private 
institutions of higher education. Ms. Giraitis, on behalf of the Coordinating Board and the 
follow-up system, began negotiations with volunteer private institutions for the exchange of 
information in a mutually acceptable format and for cooperation and collaboration in future 
follow-up studies. 
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Degree of Automation 



To the extent that negotiations for access to data bases (such as private institutions of 
higher education) extending the breadth of coverage are successhil, automated record linkages 
will replace traditional survey techniques to locate an ever increasing portion of subjeas being 
studied. . 

Funds from the Program Year 1993-94 grant were used to purchase a desktop computer 
for the Coordinating Board and a tape drive for the SOICC. These" purchases increase the 
capacity of both parties to process information requests. 

Improving the Depth of Coverage 

The Perkins Committee of Postsecondary Practitioners and higher education members 
of the Follow-Up Steering Committee initiated discussions about the use of longitudinal 
research to determine: pre-enrollment/post-exit employment gains; post-exit employment 
retention; post-exit earnings gains; and long-term career advancement. These advances await 
approval of additional funding in subsequent program years. 

Tool Development 

Dr. John Grable (President, Brazosport College), Dr. Stanton Calvert (Executive 
Director, Texas Jr./ Community College Association) and Dr. Milton Holloway (economist) 
took preliminary steps to measure the value added by training providers to their students and 
to the economy as means of calculating returns on the investment of public dollars in 
education and training. 

Dr. Susan McBride (Northeast Texas Community College) and Dr. Darlene Morris 
(Texas State Technical College System) were among the first to take issue with the crosswalk 
used to determine the degree of training-relatedness of job placements in the Program Year 
1992-93 study. The System Director, in collaboration with the Follow-Up Steering Committe, 
devised a method for continuous validation and updating of the CIP-to-OES crosswalk. 

Initial crosswalk validation was performed by community and technical college 
administrative and instructional personnel (along with Lorraine Merrick's staff in the 
Career and Technology Education Division, TEA). Each campus was supplied with a data 
disk containing all portions of the crosswalk pertaining to the programs offered thereon. 
Administrators and instructors also were provided a software package that invited them to 
recommend and justify the addition, deletion or reassignment of values in the croc<^walk. 
In addition, an entry level/career ladder relationship was created to give credit to training 
programs that prepare student for work in fields where firms traditionally under-employ 
new entrants into the business or industry. Practitioner input was used to refine the CIP-to- 
OES crosswalk for use in the 1993-94 study. 
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Continuous improvements in the crosswalk will be made as a result of a challenge process 
devised by the System Director. As employers supply occupational titles, the subcontractor 
uses the most current version of the crosswalk to compute preliminary training-relatedness 
scores. Records are returned to the institutions for internal review. Program administrators 
and instructors will be allowed to challenge the training-relatedness score assigned to any 
former student's job placement. There are three grounds for challenging a training- 
relatedness score: 

a) Upon contacting the student, the institution learned that competencies acquired in a pro- 
gram are used in an unpaid volunteer position. (For example, a former medical tech- 
nology student employ^ during the day as an accountant volunteers as a paramedic.) 

b) Upon contacting an employer, the institution learned the duties and tasks performed by 
its former student were assign&i an improper OES code. When a more appropriate code 
is assigned, a higher degree of training-relatedness can be attributed to the job placement. 
(For example, more details were learnt about the duties of a "Technician Not Elsewhere 
Classified. " That resulted in recoding the title to "Chemical Technician. " The job place- 
ment was given a "Direct Relationship " rating as an outcome for a Chemical Technology 

program graduate.) 

c) Upon review of duties and tasks involved in an occupation, there is evidence that a genu- 
ine relationship between a training program and an occupation was overlooked by all 
who participated in Validating the crosswalk. 

Challenges would be submitted to the SOICC by a deadline. Explanations would be 
reviewed by a panel of practitioners. Challenges would be stripped of personal and 
institutional identifiers to insure that the panel judged each case without consideration of 
self-interest. On a majority vote, the panel would take one of three actions: 

1) If the challenge was rqected, the training-relatedness score and the crosswalk would 
remain unchanged. 

2) If the challenge was accepted under conditions (a) or (b), the individual training- 
relatedness score would be changed (thus improving the program and institution's 
overall score on that performance measure); the crosswalk would remain unchanged. 

3) If the challenge was accepted on the grounds that the crosswalk was in error, 

• the individual score would be changed; 

• the program and institution's success rates would be recalculated; 

• the crosswalk would be updated; and 

• the software routine to assign training-relatedness scores would be rerun to give like 
credit to all individuals, programs and institutions where the identical combination of 
CIP and OES codes was recorded. 
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Any release of training-relatedness calculations prior to the deadline and panel review 
would be list&i as "preliminary - subject to institutional challenge". Recalculations with 
the updated crosswalk would be considered "official." 

Higher education representatives on the Steering Committee, an ad hoc committee of 
higher education follow-up data users, and the Perkins Committee of Postsecondary Praai- 
tioners took the lead in advising the employer survey subcontraaor on methods for analyzing 
patterns of geographic mobility in former students’ job search activities. Drs. Wolf, Adelman, 
and England helped devise ways to make fair comparisons of entered-employment earnings to 
earnings information for both the region in which the training was provided and in the region 
where employment was secured. 

Dr. Ron Hufstuttler, Vice President, East Texas State University, used his analysis of 
pilot study results to challenge the linear model of education and training pathways. He 
discovered a significant portion of his institution’s leavers transferred to community and 
technical colleges. He suggested the need to develop more sophisticated tools to determine how 
students put together their own "program” by mixing community/technical college, bacca- 
laureate, continuing education, and graduate courses with work-experiences to acquire unique 
combinations of skills and competencies required for employment and/or employment reten- 
tion in a constantly evolving labor market. 



Utilization 



Practitioners on the Steering Committee are called upon frequently to present follow-up 
information to decision-makers who have no particular training in evaluation research or sta- 
tistics. They were instrumental in ensuring greater utilization of follow-up information by 
insisting that it be presented in easy-to-understand report cards and graphical formats. 

Members of the Steering Committee who held offices in associations for higher 
education professionals used their influence to secure spots on workshop, seminar and 
conference agendas for follow-up staff and liaison. Don Perry handled arrangements for the 
TAPSOEA fall conference; Mary Korfhage handled arrangements for a Student Tracking 
Workshop in Richardson and the annual TAIR conference. These opportunities were used to 
explain follow-up and to render technical assistance to encourage wider acceptance of the 
system and the use of its data in continuous program improvement. 

Dr. John Grable arranged for a joint presentation with the System Director at the 
annual conference of the American Association of Community Colleges. Such appearances 
before national groups foster increased utilization of follow-up data by ensuring that its 
methodology generates data that can be used fairly and objectively in cross-state comparisons. 
They also set the stage for improved collaboration and cooperation among praaitioners across 
the nation to resolve common technical issues. Greater visibility - in particular after an award 
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was presented to Dr. Grable by the AACC for his innovative approach for a data-driven 
process of continuous program improvement - ensure that Texas will remain among the leaders 
in follow-up and evaluation research. 



5) Communications and Outreach 

The System Director, employer survey subcontractor and the tri-agency grant 
management team participated in professional association activities related to follow-up, 
organized three regional workshops and a statewide conference, and engaged in other 
discussions intended to increase the likelihood that data from the follow-up system would be 
used in a process of continuous program improvement. Communications consisted of: 

• negotiating system expansion; 

• rendering technical assistance; and 

• building consensus. 

Negotiations were successfully concluded for expansion of the follow-up system in 
Program Year 1994-95 for accessing the following: 

« federal civil service on file with the Office of Personnel Management; 

• employment records on file with the US Postal Service; 

• military personnel records on file with the Department of Defense; and 

• UI wage-records on file with the New Mexico Employment Security Agency. 

Additional negotiations are underway with the Oklahoma’s Board of Regents and 
Employment Security Agency. 

Limited arrangement also may be made with Alaska, Arizona, Florida, Illinois, North 
Carolina, and North Dakota. While few hits are expected from matches against these data 
bases, the arrangements will set the stage for establishing a national employment information 
clearinghouse. 

Within the state, briefings were conducted with the Intergovernmental Relations unit 
of TEC and with the Evaluations Unit of the Texas Department of Human Services. TEC 
pledged its continued support as a provider of outcomes information. DHS entered into 
negotiations to have the follow-up system collect outcomes data for the JOBS and Food Stamp 
Employment and Training Programs. DHS also is considering its role as a provider of 
outcomes information related to decreased welfare dependency among participants who 
complete workforce development programs. Negotiations have been initiated through Wind- 
ham School staff to gain access to corrections information. 



Fitud Report: Automated Student and Adult Learner Follow-Up, 1993-94 



Page 71 



ERIC 



82 



Technical assistance was rendered in the fall to practitioners at site-based committee 
meetings at five locations across the state (Amarillo, Beaumont, Houston, Lancaster, and 
McAllen). At those meetings, the System Director explained the basics of follow-up to new 
constituents from the JTPA system and public education. He relied on experienced higher 
education practitioners at each site to give practical examples of the uses they made of follow- 
up data. Each site-based committee member was charged with the responsibility of explaining 
follow-up to other practitioners at the institutions they represented and to their peers through 
professional association activities. 

By spring, once new issues had been addressed and as new activities were added to or 
planned for the follow-up system, there was a need for additional technical assistance. Three 
regional workshops were held (Houston, Dallas and San Antonio). At each, an orientation was 
conducted by the System Director to acquaint newcomers with the basics of follow-up. The 
General Session of each workshop was opened by a recognized local practitioner whose 
prestige, testimonial and endorsement added to the credibility and acceptance of the follow-up 
system. The System Director, the employer survey subcontractor, tri-agency liaison, and 
practitioners made presentations on general policy and technical issues. Representatives from 
all three constituencies, by participating together, "cross-fertilized" each other in the sense that 
they identified areas of mutual concern and came to appreciate the need to understand follow- 
up information about their partners’ programs. 

Barbara Cigainero, Executive Director of TCWEC, and Nancy Atlas, Chairperson of 
the Higher Education Coordinating Board, opened the statewide conference in Austin in June. 
The bdance of presentations focused on results of the 1993-94 studies and plans for the 1994-95 
Program Year. The conference, like the regional workshops, was successful in generating 
increased interest in and enthusiasm for expansion of the follow-up system. In addition to 
representatives from the three current constituencies, the conference was attended by: 

representatives of the Texas Department of Human Services; 

representatives of the Texas Employment Commission; 

TCWEC members and staff; 

representatives of the Governor’s Office, the Legislative Budget Board, legislative aides, 

and the State Comptroller’s Office; and 

out-of-state delegations from Oklahoma, Arkansas, Arizona, and Illinois. 

Invitations were issued to out-of-state parties as part of a larger effort to learn from and 
to build consensus among all persons engaged in follow-up activities within and out-of-state. 
To the extent that common definitions and parallel methodologies are developed, utilization 
of follow-up information will increase because the system will produce meaningful information 
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that can be compared fairly with data acquired by all the state’s workforce development 
partners and by parallel authorities in other states. To this same end, the System Direaor 
participated in the following: 

(with Dr. Jim F. Reed, employer survey subcontractor) attended a national workshop 
on performance measures and standards under the Perkins Act; 

organized an ad hoc committee of nationally prominent figures in automated follow-up 
to address issues related to data privacy and confidentiality; 

(with Dr. John Grable, President of Brazosport College) made a presentation to the 
annual conference of the American Association of Community Colleges; and 

(with Dr. Jim Reed) made a presentation to the annual conference of the Association 
of Institutional Researchers. 

With recognition of Texas’s leadership role in follow-up, national practices are more likely to 
conform to those we devise. In the long run, exercising that leadership will require fewer 
changes in Texas’s methods if ever the federal government mandates uniform procedures. 

Within the state, the System Direaor, the employer survey subcontraaor, and various 
members of the tri-agency grant management team and the Steering Committee made 
presentations at: 

mid-summer conference of Secondary Vocational Administrators and Counselors; 
TEXSIS data users’ group meetings; 

LoneStar data users’ group meetings; 
semi-annual (fall) TAPSOEA conference; 
annual conference of the Texas Placement Association; 
a student tracking seminar in Richardson, Texas; 

JTPA annual planning conference; 

mid-winter conference of Secondary Vocational Administrators and Counselors; and 
annual conference of the Texas Association of Institutional Researchers. 
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Lastly, the System Direaor was frequently called upon to explain the follow-up system 
and its methodology to: 

three meetings of the TCWEC (full council); 

Task Force on the State Strategic Plan for Workforce and Economic Competitiveness; 

three meetings of the TCWEC Committee on Performance Measures and Evaluation; 

at least one meeting per month with TCWEC staff. 

The purpose of these briefings was to ensure consistency, cooperation, and integration of tri- 
agency, praaitioner, and TCWEC goals and objeaives for the follow-up system under provi- 
sions of Senate Bill 642. 



Plans for Program Year 1994-95 and Recommendations 

Much work remains if the follow-up system is to meet the goal under Senate Bill 642 
for statewide integration and comprehensive ':.overage by the end of Program Year 1996-97. 
The first requirement will be a permanent source of adequate funding. In Program Year 1994- 
95, the SOICC enters its third and final year of eligibility for Perkins demonstration and 
capacity building funds used to date to support the automated follow-up system. A legislative 
proposal was drafted for the TCWEC suggesting that it take the lead in securing a permanent 
place in the State’s General Revenue Budget for biennial follow-up funding. Auxiliary dollars 
might be obtained through federal demonstration grants (other than Perkins) 
and through contributions from the state’s workforce development partner agencies. 



Breadth of Coverage 

Additional funds will be needed in subsequent years: 

• to expand JTPA coverage to all titles; 

• to move from volunteer pilot coverage to full statewide implementation on behalf of 
public education (also to include Adult Basic Education and Windham prison schools); 

• to include DHS JOBS and Food Stamp Employment and Training participants in 
follow-up studies; 

• to pay for increased data acquisition costs of record matching and the employer survey 
to handle increases in the number of seed records as new programs are served; 
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• to pay for increased data acquisition costs as longitudinal services are provided to all 
constituencies; and 

• to pay for increased data acquisition costs of record matching as negotiations are 
successful in establishing record linkages to additional data bases. 

Program Year 1994-95 will, i^ain, be considered a transitional year. The system will 
focus on activities designed to yield maximum gains in breadth of coverage: on-going 
negotiations with DHS for expansion of services; negotiations with New Mexico and Okla- 
homa for data sharing; and supporting activities to increase coverage for students and 
participants. In particular, as the State Legislature reconvenes in 1995, we will serve as resource 
persons to the TCWEC and member ^encies to provide documentation and backup materials 
to support legislative proposals calling for informed consent in the use of Social Security 
numbers for organizing participant information in all workforce development programs. 

Recommended Activities Moving Toward a More Mature System 

Degree of Automation 

Funding estimates for system expansion largely depend upon a decision by the state to 
move to an enhanced UI w^e-record. Until occupationally-specific data are included in 
employers’ quarterly reports, an employer follow-up survey will be required if performance 
measures are to include calculations of training-relatedness. In discussions with TEC, it does 
not appear that an enhanced UI wage-record system could be in place before 2001. While the 
employer follow-up survey will grow more efficient with economies of scale, it can never be 
as cost-effeaive as fully automated record linkage techniques for acquiring occupationally- 
specific outcome information. 

As new constituencies are served and as longitudinal data are acquired additional 
hardware purchase may be necessary - particularly to enhance data processing capacities at the 
Coordinating Board and TEC. A wide-area network subscription would improve system 
capacity and the speed of its responses to data requests. 

Depth of Coverage 

Follow-up in Texas continues as a series of snapshots of successive exiting cohorts. 
Additional funds will be needed for longitudinal studies. The table on the next page uses 
services to public education to illustrate how longitudinal studies would progress assuming that 
funds were made available for statewide implementation before the beginning of Program Year 
1995-96. 
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Tool Development 



Validation of the CIP-to-OES crosswalk must be done on an annual basis. The focus 
in 1994-95 should be on modifying the system to better serve public education’s need to 
determine the training-relatedness of job placements. As the system adds longitudinal data, 
new time-series analysis tools will be needed. As local workforce development boards are 
created, their geographic composition must be entered into the Zip code-to-state administrative 
regions crosswalk. 



Utilization 



To some extent, wider use of follow-up information will be made as the reports and 
graphical presentations improve. To that end, the word processing software currently in use 
should be updated to include more desktop publishing features. To help practitioners and 
laypersons intuitively grasp complex data, we recommend the purchase of a Geographic 
Information System - software to map outcomes by county or region. To make graphical 
presentations more engaging, we recommend the purchase of color printing or plotting 
hardware. 

The chief means of promoting maximum use of follow-up Jata, however, will remain 
our communications and outreach efforts. Continued technical assistance will acquaint ocperi- 
enced practitioners with new tools and presentation formats. So, too, will technical assistance 
bring replacement personnel and new constituencies’ researchers up to speed. 

Most important are plans to reach out to parental and community based organizations 
to explain how follow-up works and what benefits they can expea from system expansion. 
At the same time follow-up staff will be available as resource persons to legislative and 
administrative leaders. It is through such efforts that the system will take the largest steps in 
the transition from its narrow use by specialists to widespread acceptance and usage in public 
policy making. 

Finally, work should be^n by the end of Program Year 1994-95 to incorporate follow- 
up findings in SOCRATES (the automated planning model) and Texas CARES (the automated 
career in-formation delivery system). Results for community and technical colleges will be 
added first to the GEM and SCHOOLTRAIN modules of SOCRATES. By packaging these 
data within other SOICC produas; we will facilitate wider access and user understanding at 
critical junaures in both institutional and individual decision-making. 
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APPENDIX II 

FILE LAYOUT - PUBUC EDUCATION SEED RECORD 



Variable 


Type 


Length 


Start 


End 


Explanation | 


Record Type 


alpha 


1 


1 


1 


"H” « high school 


District 


alpha 


6 


2 


7 


Distria identiHer assigned by TEA 


Social Secur- 
ity Number 


alpha 


9 


8 


17 


Unique individual identifier 


Gender 


alpha 


1 


18 


18 


values in PEIMS standards 


D.O.B. 


alpha 


4 


19 


23 


Date of Birth (MMYY format) 


Campus 


alpha 


3 


24 


26 


For multi-campus districts 


VocEd 


alpha 


1 


29 


29 


Was student enrolled in a voc ed course 
during the October recording window? 


Graduation 

Type 


alpha 


2 


30 


31 


Values in PEIMS standards 


LEP 


alpha 


1 


32 


32 


Limited English Proficiency identiHer 


Economically 

Disadvantage 


alpha 


2 


33 


34 


Perkins special pops identifier 


Special Ed. 


alpha 


1 


35 


35 


Perkins special pops identifier 


Bi-Lingual 


alpha 


1 


36 


36 


Perkins speciid pops identifier 


ESL 


alpha 


1 


37 


37 


English as Second language identifier 


Immigrant 


alpha 


1 


39 


39 


Perkins special pops identifier 


Pregnant 

Teen 


alpha 


1 


40 


40 


Perkins special pops identiHer 


Grad. Month 


alpha 


2 


42 


43 


MM format 


Grad. Year 


alpha 


2 


44 


45 


YY format 


Ethnicity 


alpha 


1 


46 


46 


Values in PEIMS standards 


Last Name 


alpha 


8 


50 


57 


Truncated to tit; for SSN validation H 


Initial 


alpha 


1 


58 


58 


Middle Initial; for SSN validation 


First Name 


alpha 


2 


59 


60 


Truncated; for SSN validation 


School Year 


alpha 


2 


61 


62 


YY format 


Blank fields 


blank 

fUl 








found in positions 18, 26, 27, 28, 38, 41, 
1 47, 48 and 49 for record sizing 
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FILE LAYOUT - HIGHER EDUCATION SEED RECORD 



1 Variable 


Type 


Length 


Start 


End 


Explanation 


1 Record Type 


numeric 


1 


1 


1 


Graduate or leaver 


1 ncE 


numeric 


6 


2 


7 


Institution identifier 


1 Student ID 


numeric 


9 


8 


16 


unique Social Security number 


Gender 


alpha 


1 


17 


17 


Gender of student 


Classiflcadon 


numeric 


1 


18 


18 


Student level (freshman, etc.) 


DOB 


numeric 


4 


19 


22 


MMYY format 


First Time 

Transfer/In 

College 


alpha 


6 


23 


28 


Flags 1st time in college or FICE code of 
institution where student first enrolled in 
postsecondary ed. and training 


Type of 
Major 


numeric 


1 


29 


29 


Academic, Technical, or Tech Prep 


Major 


numeric 


8 


30 


37 


Declared major by CIP code 


Sem. Credit 
Hours (Fall) 


numeric 


2 


38 


39 




Sem Credit 
Hrs. (Spring) 


numeric 


2 


40 


41 




Credit Hrs. 
(Summer I) 


numeric 


2 


42 


43 




Credit Hrs. 
(Summer II) 


numeric 


2 


44 


45 




Ethnic Origin 


numeric 


1 


46 


46 


see CB code values 


Last Semester 
Enrolled 


numeric 


1 


47 


47 


Fall, Spring, Summer I or Summer II 


Last Year 
EnroUed 


numeric 


2 


48 


49 


YY format 


Type of 
Award 


alpha 


8 


50 


57 


Type of degree or certificate awarded 
upon graduation 


Level of 
Award 


numeric 


1 


58 


58 


Associate, Certificate, Advanced Skills 


Month 

Graduated 


numeric 


2 


59 


60 


MM format 


Yr. 

Graduated 


numeric 


2 


61 


62 


YY format 
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FILE LAYOUT - JTPA SEED RECORD 



Variable 


Type 


Length 


Start 


End 


Explanation 


Record Type 


alpha 


1 


1 


1 


Include in employer survey (yes/no) 


SDA Number 


numeric 


3 


2 


4 


Assigned by TDoC 


Grant 


numeric 


2 


5 


6 


Under what title did subject participate? 


Welfare 

Recipient 


alpha 


1 


7 


7 


Yes/No 


SSN 


numeric 


9 


8 


16 


Social Security Number 


ID Number 


numeric 


6 


17 


22 


Assigned by SDA { 


Stx 


alpha 


1 


23 


23 


Gender | 


Ethnicity 


numeric 


1 


24 


24 


Ethnic group code H 


DOB 


numeric 


6 


25 


30 


YYMMDD format 


Education 

Status 


numeric 


1 


31 


31 


TDoC assigned code for level of 
education upon entering program 


Postsecondary 


numeric 


1 


32 


32 


Upon entry, was participant attending a 
postsecondary institution? 


Grade at 
Termination 


numeric 


2 


33 


34 


Functioning grade level upon program 
termination 


Termination 

Date 


numeric 


6 


35 


40 


YYMMDD format 


Termination 

Reason 


numeric 


2 


41 


42 


Code explaining status at program end fl 


Placement 


numeric 


J» 


43 


47 


Job by OES at time of placement | 


LastOES 

training 


numeric 


5 


48 


52 


OES code of last occupationally specific | 
training intervention by JTPA 


Train. Type 


numeric 


1 


53 


54 


OJT or Classroom 


Completed 

Training 


.alpha 


1 


55 


56 


Yes/No 




numeric 


6 


57 


62 


zero fill to standard seed record length | 



Final Report: Automated Student and Adult Learner Follow-Up, 1993-94 



Page 81 



APPENDIX II 



FILE LAYOUT - HIGHER EDUCATION OUTCOMES APPENDED 



In™*™***"**” 

1 Variable 


Type 


Leng;th 


Start 


End 


Explanation 


1 Xfer HCE 


alpha 


6 bytes 


63 


68 


FICE code for transfer institution 


Institution 

Type 


alpha 


1 byte 


69 


69 


Transfer to community/techincal coU^e, 
university or health science center 


Xfer Major 


alpha 


8 bytes 


70 


77 


Major at transfer institution by CIP code 


Xfer T)rpe of 
Major 


alpha 


1 byte 


78 


78 


Academic, Technical or Tech Prep. 


Semester 

Hours 


alpha 


2 bytes 


79 


80 


Number of credit hours attempted in the 
matched term at transfer institution. 


Semester/ 

Year 


alpha 


3 bytes 


81 


83 


Last two digits of the matched year and 
semester code (always fall term) 
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APPENDIX n 

FILE LAYOUT ~ LABOR MARKET OUTCOMES APPENDED 



Variable 


Type 


Length 


Start 


End 


Explanation 


First Name 


alpha 


1 


84 


84 


For veriHcation of valid match and 
identification in employer survey 


Initial 


alpha 


1 


85 


85 


ditto 


Last Name 


alpha 


10 


86 


95 


ditto 


SIC code 


char- 

acter 


4 


96 


99 


Standard Industrial Classification of 
firm’s principle business/industry 
activity. 


Employer ID 


alpha 


9 


100 


108 


Unique firm identifier used in TEC 
records to link UI data base to contact 
infonnation data base 


Firm Name 


alpha 


35 


109 


143 


First line of company name 


Company 
Name (2) 


alpha 


35 


144 


178 


Second line of company name (if 
necessary) 


DBA 


alpha 


35 


179 


213 


Doing Business As (company name) as 
backup contact information. 


Address 
(line 1) 


alpha 


35 


214 


248 


Contact information for distributing 
employer follow-up survey. 


Line 2 


alpha 


35 


249 


283 


ditto 


City 


alpha 


20 


284 


303 


ditto 


blank 




1 


304 


304 




State 


alpha 


2 


305 


306 


ditto 


blank 




2 


307 


308 




Zip + 4 


alpha 


10 


309 


318 


ditto 


Wages 


alpha 


5 


319 


323 


Whole $ for matched quarter 


Wages 


alpha 


2 


324 


325 


Remaining cents 


Quarter 


alpha 


3 


326 


328 


Year /quarter match was found 



O 

ERIC 
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APPENDIX II 



FILE LAYOUT - EMPLOYER SURVEY RESULTS APPENDED 



1 Variable 


Type 


Length 


Start 


End 


Explanaticm 


R Response 


alpha 


1 


351 


351 


Did the employer respond to the survey? 
(yes/no) 


1 Job Title 


alpha 


25 


352 


376 


Employer-supplied occupational title 
(verbatim). 


OESCode 


alpha 


5 


377 


381 


OES code assigned to the title by the 
subcontractor. 


Degree of 
Training- 
Relatedness 


alpha 


1 


382 


382 


Results of ;q>plying the CIP-to-OES 
crosswalk to data tiles: Directly related, 
closely related, entry-level/career ladder, 
generally related or not related. 


Zip Code 


alpha 


5 


383 


387 


Worksite (supplied by employer) 


City 


alpha 


20 


388 


407 


Worksite (converted by subcontractor 
from employer supplied Zip code) 


State 


alpha 


2 


408 


409 


ditto 


County 


alpha 


3 


410 


412 


FlPS/County code (supplied by 
subcontractor conversion of employer 
supplied Zip code) 


Full time/ 
full quar-ter 
flag 


alpha 


1 


413 


413 


Did the former student/partici-pant 
employer work at least 35 hours per week 
for the full quarter for which earnings 
data are available? 
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APPENDIX III 

SAMPLE DATA SHARING AGREEMENT 



Texas Higher Education Coordinating Board 
Request for Access 
Confidential Student Information 

CUEOT/CONTRACTOR AGREEMENT 

Requestor Name Address 

Title 

Telephone _ 

Division/Co 



I am requesting access to individual student information for the following reasons: 

Justification: The Texas SOICC has been selected as the automated follow-up grant recipient to collect outcomes 
data on former students and participants of workforce development programs in public education, higher 
education and the JTPA system. The study will benefit students bv improving program plannings evaluation and 
coordination. 



Please attach a copy of specific plans developed for tising the requested information, as well as a copy of 
procedures for protecting confidentiality. 

I understand that information concerning any individual student is to be held in strictest confidence and I assure 
that procedures are in place for monitoring and protecting confidentiality of student information. 

I understand that any unauthorized disclosure of confidential student information is illegal as provided in the 
Family Educational Rights and Privacy Act of 1974 (FERPA) and in the implementing federal regulations found 
in 34 CFR Part 99. FERPA is specifically incorporated into the Texas Open Records Act as an exception to 
records which are subject to disclosiues to the public (Art. 6252-17a, Sec. 14(e) Vernon’s Annotated Code). 

In addition, I understand that any data sets or output reports I, or my representative, may generate with 
individual student data are confidential. I will not disclose to any unauthorized person any data set or reports 
which I am given or devise. I understand that I am responsible for any computer transaaions performed as a 
result of access authorized by use of password(s). 

I also understand that failure to observe thcie restrictions constitutes a "Breach of Computer Security" as defined 
in Texas Penal Code, Chapter 33 sec. 33.02,B, and that such an offense constitutes a Class A misdemeanor. 



Signature of Requestor Date 



APPROVED BY: 

General Counsel Date 

APPROVED BY: 

Assistant Commissioner for Administration Date 
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APPENDIX IV 



STEP BY STEP GUIDE 

FOR USING AUTOMATED FOLLOW-UP DATA IN 
A PROCKS OF CONTINUOUS PROGRAM IMPROVEMENT 



Overview 

To date, performance standards are composite reflections of overall program 
effeaiveness. That is, they are expressed as a single minimum level of expected outcomes that 
can be achieved through success on a variety of performance measures. For example, under 
Coordinating Board rules, a program is considered successful if 85% or more of its former 
students are placed in the labor market, continue to pursue education and training elsewhere, 
and/ or join the military. The three hypothetical programs below would be considered equally 
successfol though each met the 85% standard via drastically different combinations of 
outcomes: 





PROGRAM A 


PROGRAM B 


PROGRAM C 


Working only 


20% 


40% 


15% 


Pursuing education only 


40% 


15% 


30% 


Enter the military only 


1% 


1% 


22% 


Combination of the above 


27% 


32% 


21% 


Documented Successes 


88% 


88% 


88% 



The illustrations used herein assume that an initial examination of follow-up data for 
a program indicated its performance fell below the composite standard. While the illustrations 
below use the Coordinating Board s 85% Successful Outcomes Rule” the diagnostic procedures 
are applicable no matter what mix of performance measures are available to the education and 
training provider in documenting the effeaiveness of a program as a whole. So, too, is the 
diagnostic process applicable no matter what level of composite success is set as the standard. 

The data obtained from automated follow-up provides some but not all of the 
information necessary for administrators and service providers seeking to improve the 
curriculum and its delivery. Follow-up data are like indicators and a compass on an 
automobile’s dashboard. They tell you how fast you are going and what direaion you are 
headed. They may even provide early warning signals about problems. Standing alone, they 
do not explain why a problem exists nor what correaive aaion, if any, is necessary and 
appropriate. These data, however, suggest where to look next in a logical process of 
elimination as praaitioners attempt to pinpoint problem(s) and recommend solutions. 
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STEP ONE - Review the Formula and Figures Used in Calculating Successful Outcomes 

The Texas SOICC and participating agencies will make preliminary calculations of 
successful outcomes. The calculation by program is simple: 



unduplicated count of successful outcomes / 

' total useable records of completers and leavers 



The gross calculation is not without criticism. In part, it holds education and training 
providers accountable for factors beyond their control. Practitioners on the Follow-Up 
Steering Committee have suggested two principle considerations to be entertained in 
recalculating success rates. By using one or both in recomputations, the education and training 
provider may be able to demonstrate satisfactoiy performance. Either faaor increases the 
likelihood of a higher successful outcome ratio by reducing the denominator. 

Are some program completers unavailable for activities counted as successful outcomes? 
Obviously, deceased persons should be taken out of the calculation. Not so obvious 
would be incarcerated persons. For example, if vocational training is offered as 
rehabilitation in a corrertional facility, students/participants might complete a program 
before becoming eligible for release. Such individuals should be followed. For follow-up 
purposes, however, they should be included in the cohort based upon release date 
rather than program completion. 

The initial calculation does not take into account student intent. It assumes that the 
outcomes desired by taxpayers, administrators, service providers, prospective employers 
and the vast majority of students/participants are shared by all who enroll in a 
program. Programs, however, may attract persons fulfilling avocational and self- 
development interests. This may be especially true for those enrolled in adult 
vocational courses; however, some declared majors may have no intention of pursuing 
what others would define as desired outcomes. One could argue that such persons also 
be removed from the denominator. 



STEP TWO • Consider Collecting Supplemental Follow-Up Information 

The Automated Student and Adult Learner Follow-Up System matches seed records 
to a finite set of administrative data bases. Taken collectively, those data bases do not exhaust 
all possible successful outcomes. Great care is taken in reports to acknowledge the limitations 
and gaps in available data sources. Missing documentation of success(es) in a former student’s 
record is not labeled as "unsuccessful-” rather, all such cases are reported as "not located" If a 
program falls below the performance standard, administrators should first determine if 
supplemental follow-up is warranted. 
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Several factors should be taken into account when deciding to conduct supplemental 
follow-up. First, is there reason to expect that a disproportionate number of the program’s 
successes fall within the gap(s) in the data bases linked by the automated system? The chief 
gaps in the follow-up system’s coverage do not effect all programs uniformly. 

The Texas Unemployment Compensation Aa does not cover self-employed 
entrepreneurs. Some programs train persons for occupations where the 
opportunity structure is skewed toward self-employment. Many cosmetologists, 
real estate agents, and musicians, for example, are not included in TEC’s wage 
record system. Exemption from the Unemployment Compensation Act for 
some farm workers has similar effects on follow-up for agricultural programs. 

Students from institutions on the perimeter of Texas may find jobs or continue 
their education in an adjacent state. Until data sharing agreements with 
surrounding states are negotiated, the automated system can not document those 
successes. 

Unpaid volunteer work is not covered by the Unemployment Compensation 
Act. Paramedics and emei^ency medical specialists, for example, may be using 
training received even though their volunteer work can not be documented by 
the automated follow-up system. 

Opportunities in some fields for advanced training and education as the logical 
continuation of lower level programs offered at Texas’s public institutions might 
be available only in out-of-state or at private institutions not yet linked to the 
automated follow-up system. 

In each case, the probability of discovering additional outcomes may be high. In 
deciding to condua supplemental research, weigh your estimate of the likelihood of additional 
hits against the availability of resources for the data collection effort and your confidence in 
the accuracy of contact information in alumnae files. 

If supplemental research uncovers additional successful outcomes for those not located 
by the automated system, the new data should be reported to the Texas SOICC in standar- 
dized format (to be set in collaboration with the agencies) by the deadline (to be announced 
annually) with sufficient backup documentation for audit and verification. After the sub- 
mission deadline passes, the SOICC will calculate the revised "successful outcomes" score for use 
in all official reports. 
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STEP THREE - Disaggregate Available Follow-Up Data by Type of Outcome 

Assume that either supplemental follow-up was not conducted or that the additional 
documented successes did not suffice to move performance above the standard. The next step 
is to disaggregate the data by type of outcome. Separate successhil outcome rates for continued 
pursuit of education and training, job placements, and entered the military will be available. 
Failure to meet a composite performance standard may be attributed largely to poor 
performance in one but not all of the successful outcomes categories. 

At present, standards are not published separately for each type of successful outcome. 
The sequence in which the components of success should be examined can be prioritized 
according to the training provider’s educated guesses about what may have caused the shortfall 
and/ or where corrective action is most likely to be fruitful. In the absence of disaggregated 
external standards, it is the task of the education and training provider to determine what mix 
of outcomes should have contributed to the composite success of its programs. The antici- 
pated mix may be discerned from a mission statement or from the historical records sur- 
rounding creation and initial implementation of each program. When the program was created, 
who collaborated on curriculum development: military specialists? business and industry (e.g., 
through the DACUM process)? admissions or training-providers from institutions offering 
articulated programs? What was the program primarily designed to achieve: preparation for 
military service? imparting competencies for the workplace? laying the foundation for 
additional education and training? Actual outcomes must be compared to self-imposed or 
hypothesized expectations to determine more precisely where (the) problem(s) occurred. 

In some cases, self-imposed performance standards (or expectations) may be available. 
Did the training provider set program performance standards during a self-study for an accredi- 
tation visit? Were targets set in corrective action plans submitted in prior years for the 
improvement of subpar programs? Were performance levels promised or implied when 
applying for program funding or in recruiting students/participants? 

In other cases, self-imposed performance standards must be inferred from other 
materials: 

• How do outcomes for the program’s most recent cohort compare with prior 
cohorts? Has there been a significant downturn in the proportion of successful 
outcomes on one of the component measures? Is the downturn more noticeable 
on one component than in the others? Has any downturn accelerated? (Have 
successes in one aspect remained constant or failed to accelerate as 
improvements have been made on other component measures?) 

• What did the students expect when they entered a program? What can be 
distilled about implied program expectations from guidance materials, counseling 
notes or, preferably, local empirical information about student intent? 
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• How are comparable programs performing elsewhere? In particular, if the 
program under examination emulates another, how is the emulated program 
performing on each component measure? If rooted only in theory, what does 
the related body of literature suggest about expected outcomes? 



STEP FOUR - Analyze Externalities 

Externalities must me analyzed to determine if expectations on each component 
measure were realistic. The list below is suggestive and is not intended to be exhaustive. 

Entering the Military as an Outcome 

Is there a decreased need for personnel in the armed services? Has the military in 
general decreased its recruitment efforts and/or made entrance requirements more stringent? 
Have bases near campus closed or downsized? Have any or all of the branches suspended or 
decreased their recruitment efforts on or near campus? Has military service become less 
attractive because increases in compensation and benefits have not kept pace with those for 
civilian jobs? Has military service become less attractive because changes in international affairs 
and American foreign policy have increased the perceived risks? Has there been a decline in 
the need for personnel in those specific military subspecialties for which the training program 
was designed? Have the required competencies changed in the military subspecialties the 
program was designed to address? 

Follow-up data, per se, can not answer these questions but should suggest when it is 
appropriate to ask them. Local recruiting officers and defense conversion/transition specialists 
should be consulted. Their answers will suggest appropriate remedies for programs designed 
chiefly as preparation for military service. 

If no faaors can explain lower than expeaed military enlistments, the activity most 
likely to improve results would be facilitating increased recruitment on campus and/ or 
increased communications between the institution’s placement office/ counselors and 
military recruiters. 

If the competencies required in the military subspecialties addressed have changed, 
militaty specialists should be consulted in updating the curriculum or revising it to 
emphasize emerging and shifting military subspecialty personnel needs. 

If changes simply indicate declining opportunities for military enlistment: 

shift the curriculum to a civilian employment emphasis in conjunction with 
defense conversion and transition plans; 
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emphasize academic and technical components which are most likely to articu- 
late with more advanced training and education available elsewhere; or 

as a last resort, suspend or terminate the program. 

Continued Pursuit of Education and Training as an Outcome 

Have job opportunities for persons with more advanced awards decreased (thus making 
the pursuit of additional education and training less attractive)? Has renumeration for advanced 
awards decreased in absolute or relative terms? Labor market data and forecasts are available 
from Quality Work Force Planning Committees, the Texas SOICC, the Texas Employment 
Commission, Bureau of Labor Statistics, as well as labor, economics and educational journals. 
If they indicate more advanced degrees cease to provide employability advantages in a training- 
related field, the curriculum might be revised to emphasize the skills and competencies needed 
for labor market entry immediately upon program completion. 

If continued pursuit of education does provide employability advantages, other ques- 
tions should be asked. Within the region or state, do opportunities exist outside the institution 
for additional training and education related to the program being examined? Are articulation 
agreements in place with all other institutions offering opportunities for continued education 
and training in related fields? How recently were the articulation agreements negotiated or 
revised? Are incentives in the articulation agreements sufficiently strong to encourage program 
completers to seek additional education and training at the cooperating institution? Do 
student perceptions of duplicative course requirements at the cooperating institution discourage 
them from continuing? How are comparable programs doing elsewhere? 

Assuming that sound, inviting articulation agreements are in place, are program 
participants aware of the opportunities at the cooperating institution? Is articulation 
information available and distributed in hardcopy? in automated form? readily available in the 
library, counseling office and/or student center? Do program directors, instructors and 
guidance counselors inform students about articulation opportunities and assist them in seeking 
admissions at the cooperating institution with credit for coursework already completed? Are 
representatives of the cooperating institution active on campus recruiting program completers 
to continue their education and training? 

Apart from formal articulation agreements, does the program adequately prepare its 
students/participants to succeed at the next level? Do ACT/SAT scores indicate a larger than 
anticipated number of program completers were not fully prepared for admission to selective 
institutions of higher education? Does the required remediation (as a result of low TASP 
scores) discourage a program’s graduates from continuing their education and training at public 
institutions of higher education with open admissions policies? Do "electronic transcript” 
comparisons indicate that a program inadequately prepared its students for more advanced 
education and training offered elsewhere? Are a program’s students dissuaded from seeking 
admission to institutions where preceding cohorts have not performed well? Are the costs of 
pursuing additional education and training beyond the means and resources of your former 
students? Are subgroups under financial or family pressures to enter the labor market quickly? 
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Again, suspension or termination of a program because of low transfer rates would be 
a last resort. Preferable solutions would be to increase and improve articulation agreements, 
increase dissemination of articulation information, facilitate transfer institution recruitment and 
assist students with their applications for transfer. Where necessary, the curriculum could be 
revised to emphasize training for immediate entry into the labor market or preparation for 
related emerging disciplines. Lastly, the curriculum could be changed in consultation with 
representatives from the next level of education and training to ensure the competencies 
imparted by a program prepare its graduates to meet admissions requirements and to succeed 
at the transfer institution. 



Labor Market Outcomes 



Several factors could explain a program’s lower than expected job placement rate. Chief 
among those factors would be decline in occupational employment opportunities related to 
training - that is, anticipated job openings did not exist. Demand occupation forecasts are 
available in the form of the State List of Priority Occupations, Regional Quality Work Force 
Plans, JTPA Service Delivery Area Plans, and supporting documentation supplied to the Coor- 
dinating Board to justify funding of new programs. When proper procedures are followed, 
these demand forecasts and occupational targeting strategies are data-driven and conscientiously 
validated through collective regional wisdom and the checks and balances of educator and 
employer interests. When the model is followed conscientiously, programs delivering training 
and education related to the targeted occupations should result in high placement rates and 
long term retention. In particular, wages at entered employment and the rate of training-related 
placements should be high. 

Low placement rates could indicate several possibilities. Were the data-driven 
predictions of the planning model ignored? Were some programs insulated from the process 
because their providers and supporters were entrenched behind outdated demand information, 
position and political clout, or non-representative anecdotal information? Was student interest 
in a program misinterpreted or misrepresented as an indicator of employers’ demands for 
skilled workers in a related occupation? Did local business and industry representatives 
exaggerate occupational employment demand to ensure an oversupply tnat would depress wage 
demands? Was the planning model ignored or manipulated to preserve some programs in the 
face of declining or virtually non-existent occupational employment demand? If so, low 
placements validate the soundness of the planning model and should persuade planners and 
administrators or business and industry to refrain from ignoring or manipulating the process. 

A second possibility is that, while modest demand existed, several education and 
training-providers independently used the demand figures to justify their program offerings. 
In the absence of coordination among pro^riders of similarly targeted programs, the collective 
supply exceeded demand. Closely related is the possibility that while modest demand forecasts 
justified limited enrollments, demand information was not translated into enrollment ceilings 
and cautious career advising. That is, more students/participants were recruited and graduated 
than could be absorbed given the level of demand forecasted. 
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The planning model, even when validated with regional wisdom, is not infallible. The 
model relies on interpolation from historic data and is not a crystal ball. In some cases, 
unforeseen or unprecedented events will result in lower than expected job openings. Such 
events, however, can themselves be docun^nted and analyzed. Their impact will be evidenced 
in the next wave of industrial and occupational employment data (and, once the system is fully 
implemented, in the follow-up data) used in the planning model. Where earlier occupational 
demand forecasts from the model were too high, the decision to maintain, suspend or ter- 
minate a related program can be guided by: 

rerunning the planning model using the most recent data; and 

exercising regional wisdom and locally available empirical data to determine if the 

lower than expected demand for occupational employment was a temporary aberration 

or an indication of a long-term downward trend. 

The other distinct possibility is that the forecasted job openings did exist but the 
positions were not filled by graduates from the related training program. The TEC job bank, 
local personnel agencies and the classified adds can be analyzed for additional evidence that the 
forecasted job openings existed. Local prospective employers can be consulted to determine 
why the positions were not filled by program graduates. In determining the latter, there are, 
again, several possibilities to cover. Each possibility may suggest a slightly different remedy. 

If the program’s graduates applied for but were not hired for open positions, was it 
because the prospective employers found them lacking requisite competencies? If this is true 
then the service provider should ask prospective employers to collaborate in reviewing the 
curriculum and related competency assessment techniques. How recently was the curriculum 
last reviewed by a focus group (e.g., DACUM process)? Have the employers’ expected compe- 
tencies changed since they were last given a chance to help shape the curriculum? Was business 
and industry adequately represented on the last focus group? Were the firms most likely to 
hire invited to send representatives? Did business and industry send representatives with the 
appropriate expertise? Were the business and industry representatives aggressive in providing 
input? Were business and industry suggestions ignored in favor of educator opinions or resis- 
tance to changing traditional practices? Were authentic assessment instruments developed to 
measure the expected competencies? Was program completion tied to demonstrated compe- 
tence to the level of employer satisfaction? Did a disproportionate number of graduates fail 
to take or pass qualifying examinations for admission into the licensed and regulated 
professions? 

t 

If, after consulting business and industry, it appears that the curriculum and assessment 
practices were acceptable, review institutional records to determine if students/participants 
exited too early. For example, the full range of competencies necessary for employment may 
be reflected in the standards for awarding an associate degree while the less demanding cri- 
terion for a certificate award was too low to confer employability advantage. If that is the case. 



Final Report: Automated Student and Adult Learner Follow-Up, 1993-94 



Page 93 



106 



the training provider should look beyond the curriculum to other aspects. Were students 
adequately counseled and motivated to persist through to the more advanced award? Were 
students accorded the support necessary to help them persist? Here the remedy may have less 
to do with curriculum revision than the redeployment of resources into student retention and 
persistence efforts. 

If the competencies and assessment practices of the training provider square with 
employer expeaations and if students/participants persisted to receive the appropriate 
credentials and still did not fill existing openings, additional possibilities should be examined: 

Were the training-provider’s job placement aaivities sufficient? Did the placement 
office stay in communication with prospective employers? Were job notices posted 
with the placement office? Were students aware of and take advantage of placement 
services? What was the ratio of students to placement officers? 

If students were aware of and adequately trained for announced job openings, did the 
structure of employment pose additional barriers? Were students provided adequate 
information about the entry level wages they could expect if they completed a 
program? Were students too discouraged to apply because jobs were posted at lower 
than expected wages? Were employer practices constrained by ethnic, gender or special 
populations considerations (either in the form of historic discrimination or affirmative 
action mandates) that worked to the disadvantage of the program graduate mix? 

If program graduates applied for available positions, were they passed over in favor of 
applicants trained elsewhere? In the opinion of the prospective employers, does a 
competing training-provider do a better job of preparing students/participants for the 
workplace? In what competencies were the competing training-provider’s graduates 
better prepared? What can the program do to meet or exceed its competition? Has a 
prospective employer been "soured” on a program because of dissatisfaction with an 
individual graduate from an earlier cohort? Would the dissatisfied employer’s interest 
in a program’s graduates be restored if the unsatisfactory individual was retrained under 
the provisions of a graduate guarantee? If all future graduates were guaranteed to be 
competent? 



STEP FIVE - Disaggregate the Follow-Up Data by Type of Student Served 

While the mission of most publicly funded programs generally is to serve all who need 
and desire education and training, we recognize that not all students learn at the same pace or 
in the same manner. While a sound curriculum may be externally validated, its delivery must 
be tailored to the needs and learning styles of individuals. When a program fails to meet a 
composite standard for successful outcomes, the shortcomings may be more in evidence for 
subpopulations in the student mix. 
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Contained either in the seed records or local data bases are back^ound and 
demographic variables which could be used in diss^regating and explaining outcomes. The 
following items are generally fl^^ed in each record: gender, ^e (or ^te of birth), ethnicity, 
economically disadvantaged, English as Second Langus^e, limited English proficiency, migrant, 
single/teenaged parent or displaced homemaker, special education or other factors that may 
suggest barriers to persistence, the need for tailored delivery or special counseling and support 
services. The seed records with appended outcome data also can be linked back to local data 
bases which may contain more detailed information on a student/participant’s financial 
situation, intent, grade point average, pre-admission standardized test scores, and aptitude or 
interest profile. 

Where a program is generally successful in serving the majority of those enrolled but 
fails to serve all subgroups equally well, overall success may be achieved by concentrating on 
tailoring delivery to special needs: 

adjust the pace of curriculum delivery; 

offer additional tutorial and mentoring services; and/or 

use or develop instruaional materials more likely to engage the interest or fit the 

learning style of the identified subpopulation. 

STEP SIX - Review Follow-Up Data for Other Education and Training-Providers 

In all the recommendations above, the emphasis is on turning to outside sources for 
more information to better pinpoint problems and their causes. Assume the education and 
training provider and those consulted agree that an existing problem is not so insurmountable 
to warrant program suspension or termination. While those consulted outside the education 
and training system may help in understanding the nature of the problem, they may lack the 
expertise or experience to provide constructive suggestions for program improvement. The 
search for solutions, however, can be guided by follow-up data from other programs. 

In addition to identifying problems, the follow-up system can help identify best 
practices potentially worthy of emulation. For any given problem, there probably is no 
shortage of theories or marketing hyperbole touting one solution over others. Hard data, 
however, can help identify programs or practices that work. The training-provider that 
examines only its own program performance overlooks a rich source of information that could 
be used in developing concrete action plans for improvement. 

Here a note of caution should be inserted. While follow-up data can indicate what 
programs achieved better results, the education and training provider should not assume 
automatically that "best practices” used elsewhere will work under all circumstances. When 
reviewing follow-up data from other service providers, ask the following questions: 
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Did the program that achieved superior results have more resources at its disposal? 

Did the program that achieved superior results operate under more favorable 
circumstances? Was the labor market served more robust (growing or at least insulated 
from comparable downturns)? In the immediate vicinity were there more opportunities 
for continued pursuit of education or military service? 

Did the program that achieved superior results serve a different population mix? Were 
program admissions more selective? Were program participants ‘less likely to need 
remediation (based, for example, on in-coming standardized test scores)? Did the 
program serve a smaller percentage of persons in special populations (harder to serve/at- 
risk) categories? Did the program serve a smaller percentage of persons likely to face 
gender, ethnic, or age discrimination in the marketplace. 

In developing concrete action plans for program improvement, education and training- 
providers should look to more successful programs offered under comparable circumstances 
to a comparable student mix. (See STEP FIVE, supra.) 



STEP SEVEN - Continuous Review 

The illustrations above assume that a program failed to achieve a composite 
performance standard. All programs, however, need to be kept current and engaging. 
Professionalism among educators and trainers welcomes feedback on performance. Beyond 
meeting minimal standards developed by others, they compare their current performance to 
what their previous cohorts of students/participants achieved and to their self-imposed 
standards of excellence. While they may not be required to submit corrective action plans, 
they will constantly review their own role in curriculum development, instructional materials 
selection, delivery methods, assessment and grading practices. Follow-up data should be widely 
shared and frequently discussed not only by program planners and administrators but also by 
division and department directors with the instructional staff, counselors, and placement 
officers as part of a total quality management approach to serving customers. Follow-up data 
can provide early warning signals of declining (albeit technically acceptable) performance and 
point to better practices worthy of emulation. 

Conscientious professionals impose more than the minimal standards upon themselves. 
This is particularly true with respect to labor market outcomes. Current standards count all 
jobs alike when calculating successful outcomes. While standards have not yet bee i set 
officially, more stringent definitions of "success” are being proposed in agency and statewide 
strategic plans. 
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Among the more stringent measures are: 

• Job Placement 

Under current practices, a job held after program completion/termination is counted 
as a success even if the job was held prior to or during program enrollment. Pre- 
completion/termination and post*completion/termination comparisons could opera- 
tionally define ” placements” in a way that more appropriately attributes successful labor 
market outcomes to education and training programs. 

• Job Quality 

Under current practices, part-time, low wagp, temporary and "dead end" jobs count 
alike in calculations of successful outcomes. More stringent standards would include: 

• job retention among incumbent workers who sought training to update their 
skills in the face of changing occupational employment demands and long term 
job retention among all students seeking employment security; 

• wage at entered employment, full time status and benefits coverage can be used 
to operationally define "quality employment" \ and 

• pre/ post training wage gains among incumbent workers, and long term wage 
gains and/ or career advancement (over several longitudinal waves) among all 
former students/paiticipants. 

• Training-Related Placements 

Under current practices, jobs unrelated to training count the same in calculations of 
successful outcomes as do training-related placements. The purpose of the planning and 
budgeting process as well as of guidance systems, however, is to wisely channel both 
public funds and individual resources into preparation for quality employment. They 
target occupations, fund related programs accordingly and help students/participants 
make sound career decisions in the expectation that training-related placements will 
meet other quality indicators listed above. 

The SOICC will collect additional data in anticipation of revised standards as 
conceptual definitions are operationalized and revised standards are phased in. Professional 
educators and trainers, in good faith and in the spirit of professionalism, will "get ahead of the 
curve" to evaluate their programs on these more rigorous measures brfore minimal standards 
are announced. They will do so because they perceive the intrinsic value represented by those 
measures and in their extrinsic value as indicators of service to their customers: 
students/participants, taxpayers, economic developers, and prospective employers alike. 
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APPENDIX V 



USING FOLLOW-UP DATA IN PLANNING QUALITY SERVICES 
The JTPA Model 

Where occupational skills training is provided, JTPA planning rules require they be 
targeted to demand occupations which meet quality criteria set by the area’s Private Industry 
Council. Title IIA and Title III plans are submitted to TDoC for app'roval. In reviewing 
plans, the TDoC’s state labor market analyst compares submissions to planning guidelines and 
to data-driven forecasts generated by SOCRATES, the automated planning model. Some lati- 
tude is allowed for infusion of local wisdom. In some cases, that latitude is stretched. The state 
labor market analyst may recommend deletion of questionable or unsupported occupations on 
preliminary lists. In some cases, the labor market specialist advice is ignored or over-ridden. 
In other cases, participants are granted special exemption to enroll for occupational skills 
training in fields outside the target list. 

Follow-up data should be used to validate the planning model. SD A administrators are 
advised, before submitting each year’s plan, to review their prior year’s performance in light 
of their occupational targeting decisions. If the planning model and the state labor market 
analyst’s recommendations are sound, then one would hypothesize the following: 



Participants received training related to: 


Predicted Rate of Successful Labor Market Outcomes 


Targeted occupations approved by LMI 
specialist 


Very high (at or above performance standards or high 
enoi^h to earn incentives) 


Occupations recommended for deletion 
from list by LMI Specialist 


Questionable 


Occupations not on target list 


Low 



Where higher than expected success rates are achieved by those trained for occupations 
recommended for deletion by the state LMI specialist, the Private Industry Council should 
review the minutes of its meeting where the specialist’s recommendations were rejected. The 
PIC should articulate its reasoning and forward information about exceptional outcomes to 
TDoC for possible revision of planning guidelines. The same should be done by case managers 
where special exemptions were granted for training outside the tai^et occupation list resulted 
in exceptional outcomes. 

If, however, results are as hypothesized, the PICs should use the information to curb 
unwarranted deviation from the planning model. SDA administrators also can use the 
information to eliminate most special exemptions for training outside the target list. 
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Quality Work Force Planning 



The planning model that evolved in the JTPA system was automated by the SOICC 
through the SOCRATES project and adapted for use by Quality Work Force Planning 
committees. While the QWFP committees have a broader mission than JTPA SDAs, the 
logical planning sequence is the same. QWFP committees infuse SOCRATES with regional 
wisdom when developing regional taiget occupation lists and identifying a small subset of 
critical occupations. As in PIC planning of JTPA services, there are the potential abuses and 
misuses in QWFP planning. Part of the annual cycle should include the review of follow-up 
data (as above) to validate the logic used when substituting regional wisdom for data-driven 
forecasts. . 

In particular QWFP committee members should direct the regional LMI operator to 
prepare an annual report comparing placement rates, training-related placement rates, average 
quarterly earning, and the rate at which former students exit the region to find employment 
or to enroll in an institution of higher education. The report should compare outcomes for 
each of the region’s programs related to target occupations with: 

a) the composite success rate for all education and training programs offered in the region; 
and 

b) SOICC-supplied information on statewide outcomes. 

Public Education 

The planning model also is used to generate TEA staff recommendations to the State 
Board of Education for the State List ot Priority Occupations. Board members are free to 
substitute their own logic and to persuade fellow members to add occupations to the list. We 
recommend that TEA staff prepare an annual report as part of the State List of Priority 
Occupations selection process. The report should compare success rates for programs related 
to the prior year’s list with overall statewide success rates. In particular, success rates of data- 
driven occupations on the State Priority List should be compared to those moved 
independently for adoption by one or more Board members. 

Higher Education 

Before adding new programs, institutions of higher education are required by the 
Coordinating Board to submit evidence that their current programs meet success rate stan- 
dards. Follow-up data are used to document success rates. Extensive use is made by the 
Coordinating Board when evaluating institutional effectiveness and in conducting site- 
evaluations. Technical assistance may be required as new features are added to automated 
follow-up or as turn-over occurs in the institutional research offices. 
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A Self-Correcting Planning Model 

The automated planning model, SOCRATES, is based on economic theory and sound 
forecasting practices. Nonetheless, it relies on trend data and the application of current 
structures. Where technology or changes in labor inputs affea an industry, SOCRATES may 
not reflect those most recent impaas. Recommended weights, based on post facto regression 
validation techniques, are assign^ to each variable used in the formulas for predicting future 
events. Forecasts always involve an "error term” or a "confidence interval." That is, false 
precision in forecasting is avoided by expressing projections in terms of an expected range. 
Models are continuously revised in order to reduce the error term or narrowing the range 
within which outcomes are expected to fall. Models are revised by adding (or deleting 
variables based on empirical evidence of their explanatory powers and/or by reassigning 
weights to existing variables as more is known about their interactive effects. 

The SOCRATES planning model is designed to determine the appropriate balance 
between the demand for and the supply of skilled workers for occupational employment. 
Heretofore, the model has been based primarily on demand information. Follow-up adds 
important supply-side information useful in making corrections to the model. The graphic on 
p^e 102 depicts how follow-up information will be added to SOCRATES to reduce the error 
term and improve the confidence users have in its forecasts. 

Geographic Evaluation Model 

The Geographic Evaluation Model provides a standard narrative format is fleshed out 
with empiri^ data for any configuration of counties selected by the operator. These 
narratives give planners a thumbnail sketch of the region configured. Competency 
levels among a region’s labor force are a critical consideration in economic development 
and planning. Heretofore, education and training information in the narratives has been 
derived from census data that may be severely dated before they are released officially. 
Follow-up data on the most recent cohort of program completers can be added to 
GEM narrative reports on an annual basis to provide a better picture on the supply 
side of the employment equation. 

The Industry Evaluation Model 

The Industry Evalutation model forecasts growth in occupational employment by 
industry sector. It is based on current demand and historic trands information. 
Previously available data did not differentiate between the demand for experienced 
workers and new labor market entrants. Because the automated follow-up system 
collects employment information from the UI wage records by industrial code for the 
prior year’s program completers, it may be possible to better forecast the demand for 
new labor market entrants. 
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The Staffing Pattem/Matrix Statistical System 



The Matrix Statistical System breaks industry employment growth projections into 
occupational staffing patterns. It forecasts occupational employment demand by 
allocating base year industry employment projections developed by TEC using the 
annual Occupational Employment Survey. Heretofore, occupational employment 
projections have been based on statewide data derived from employers’ survey 
responses. Like the INDEVAL model, MATSTATS currently does not differentiate 
between the demand for senior workers from demand for new labor market entrants 
for each occupation. Since the employer survey collects worksite information for the 
most recent cohort of program completers, forecast can be tailored to regional 
employment conditions and practices. 

OEM/CIDS 



The Occupational Employment Model/ Career Information Delivery Model provides 
detailed infofmation about the conditions of employment for each OES coded 
occupation. Heretofore, it has been based on statewide labor force data which includes 
both senior incumbent workers and new hires. Follow-up information will give 
program participants sufficient information to form reasonable outcome expectations 
for themselves based on regionalized data for their peers who exited the education and 
training pipeline most recently. 

CROSSWALK 



The Crosswalk includes a matrix for determining the relatedness of job placements (by 
OES code) to training received (by CIP code). The follow-up system, by devising a 
process for continuous validation, has contributed significantly to the usefulness and 
acceptance of the CIP-to-OES crosswalk. Other crosswalks may be enhanced or added 
as the result of follow-up activities: Military-to-Civilian Job Titles, Private Sector-to- 
Federal Job Titles, TEA-to-CIP codes, Zip-to-Operator Configured Regions. 

SCHOOLTRAIN 



The SCHOOLS and TRAINi ng module currently lists only the availability of 
programs by institution. Follow-up activities can add information about the track 
record of each program: persistence and graduation rates, job placements, training- 
related placements, average entered employment quarterly earnings, geographic 
mobility in job search, and continued pursuit of education and training. These 
additional variables provide a better basis for participant career decision-making, 
training provider and program selection. 
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The Proposal Process 

The Request For Proposal (RFP) process is used by JTPA SDAs, school distrias and 
institutions of higher education in awarding contracts and grants, at various stages in 
curriculum development, and in the selection of instructional materials. Careful selection of 
service providers, grant recipients, subcontractors and vendors should result in improved 
performance. Where more than one proposal for participant interventions is received in a 
competitive process, it may be difficult to determine which proposal best meets the need unless 
the RFP is carefully written. Responses may be clouded with unsubstantiated claims of 
effectiveness, marketing hyperbole (such as claims to be "an exemplary program"), and/or 
unrepresentative anecdotal information. An ill-conceived RFP may require only a statement 
of work, provider credentials, a budget, and disclosure of debarment. 

Outcome information should play a latter role in the RFP process. A carefully written 
RFP for participant intervention should include the following: 

a statement of the desired outcomes; 

a statement regarding eligibility criteria for the subpopulation(s) to be served; 

an indication of minimum acceptable performance standards (in terms of outcomes) for 
each subgroup served. (In particular, how will the independent effects of the 
proposed intervention be separated from antecedent, concurrent, and/or 
intervening effects of other variables in order to attribute outcomes or a portion 
thereof to the services provided?); 

a request for documentation (in standardized format) that the bidder met performance 
standards for each targeted subgroup when delivering the proposed services in the past. 

Where the RFP is designed to establish new services or programs, bidders will not have 
performance histories. The contracting party might not know what expected performance 
levels are reasonable nor have they a basis for forecasting variance in outcomes among 
subgroups. In such cases, bidders should be asked to address the following questions: 

V^at outcomes would the bidder define as successful outcomes? 

What data sources should be tapped to document successful outcomes? How would the bidder 
propose calculating performance? 

To what performance standards on those measures is the bidder willing to be held 
accountable? 
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On what basis are the bidder's performance promises made? 

documentation of performance by the bidder in delivering the proposed service 
to a different participant mix; 

documentation of bidder performance across the board for all services delivered; 

documentation of performance by other providers using the same approach and 
materials in serving the anticipated participant mix; or 

theory, assumptions and speculation. 

Would the bidder be willing to enter into a performance-based agreement that includes 
penalty for failure to meet promised performance levels (perhaps offset with bonuses or 
incentives for exc^ing thernj? 

If the proposal is for multiple years, will the bidder accept a contingency clause calling for 

termination of the agreement if at specified intervals, promised performance levels are not 
met? 

Bidders who demonstrate an understanding of the conneaion between proposed 
services and expeaed outcomes are more likely to meet specified goals and objectives. All 
other factors being equal, those having a solid basis for making performance promises should 
be given preference as should those willing to enter into performance-based agreements. 

RFPs for pilot and demonstration programs often anticipate subsequent rounds of 
funding for program expansion and wider implementation. It is particularly tempting to invite 
proposals for "exemplary programs: While proposed programs may be innovative, unprece- 
dented, or experiniental, evidence of performance should be required before subsequent rounds 
of funds are distributed for program expansion and/or wider scale adoption. Where awards 
are made in the absence of sound interim performance measures, ineffective programs may be 
perpetuated and emulated not on their merits but simply by circular reasoning. A successful 
bidder may use the fact that an "exemplary program" proposal was accepted in the first funding 
round jw rhetorical evidence in subsequent proposals that the program or services are worthy 
of continued funding. In letting successive awards, those requesting proposals should build a 
. performance evaluation process - relying on a predetermined follow-up design - into each 
interval in the funding cycle. 

Where the proposed outcome measures coincide with those already calculated for 
other programs, contracting parties should set aside sufficient funds and arrange with the 
SOICC to be included in the automated follow-up system. This will ensure objectivity and 
standardization while eliminating any potential duplication of effort or excessive burdens 
on outcomes data providers. 
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USING FOLLOW-UP DATA IN CAREER GUIDANCE SYSTEMS 



Guidance systems act like a series of sequential filters leading a student or adult learner to distill the 
career pathway(s) which is (are) most likely to result in career success. The first two filters compare interest and 
aptitude inventories to a duties and task list and conditions of occupational employment. These comparisons 
are used to eliminate inappropriate options which fit neither the decision-maker’s interests or aptitudes. 

Once the list of options has been winnowed to a range of appropriate options, the next filter helps the 
decision-maker determine which offer the highest probability of suitable employment. It is at this stage that 
follow-up improves the selection model. Because occupational data are collected by worksite on the cohort that 
most recently exited the training pipeline, demand can be forecasted by region for entry level workers. This 
provides much more detailed and pertinent information than statewide forecasts for all levels of occupational 
employment (which combine entry level and senior incumbent workers). Having narrowed the range of options 
to those most likely to offer gainful employment, the list can be further reduced by using follow-up information 
to rank entry level wages and prospects for full time employment for each occupation. 

Assume that the list after successive filters still conuins more than one option. The next filter allows 
the decision-maker to estimate the amount of training required to enter each occupation. The first st^^e of this 
filter compares the decision-maker s current level of education and training to the competencies required for 
successful occupational employment. The filter generates a list of additional education and training to pursue. 
In the next stage, this filter tells the decision-maker which institution(s) in the region (if any) offer the requisite 
programs. 

Assume that more than one institution in the region offers the required education and training. Follow- 
Up helps in choosing among service providers by informing the decision-maker about the track histories 

program: On average, how long did it take previous cohorts to complete the program? If I complete 
only a portion of the program, what kind of outcome can I expect? What percentage of program completers got 
jobs related to their training? How did the entered-employment wages compare for completers of competing 
programs? How well did the competing programs serve persons of my gender, ethnicity, or other shared charac- 
teristics which might effect employment opportunities? How do the long-term results compare (e.g., wage gains, 
job retention, and career advancement) among completers of competing programs? In a labor market that stresses 
lifelong learning, which program best prepared its completers to pursue additional education and training? 

In addition to providing follow-up services, the SOICC is responsible for: automating a career 
information delivery system (Texas C.A.R.E.S.); developing and improving SCX:RATES (the automated planning 
model); conductii^ Improved Career Decision-Making seminars; and producing and distributing occupational 
employment infor-mation. Efforts are coordinated across all these activities to ensure consistency in definitions 
and file structures. That way, relevant data can be transported between the automated systems. By using shared 
data in automated systems and publications, the information delivered to planners, administrators, counselors and 
students by the SOKJC and its client agencies/programs is consistent. 

Until the automated career information delivery system is perfected and installed across the state, 
researchers at each campus/ training provider site are advised to share the information they receive about program 
effectiveness with recruiters and guidance counselors. 
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COMMITTEE MEMBERSHIP LISTS 
Amarillo Site-Based Committee Membership List 



Deborah Pickering* 
Assistant Director 
Panhandle QWFP 
Plaza n, Suite 1020 
Amarillo, TX 79101 


Teresa Isbell 
Institutional Research 
TSTC - Amarillo 
P.O. Box 11197 
Amarillo, TX 79111 


Marvin Hart 

Career and Technology Ed. 
Amarillo ISD 
7200 m-40 West 
Amarillo, TX 79106 


Corky Lockmiller or Pam Fogo 
Tech Prep Coordinator 
Hereford ISD 
711 E. Park Ave. 

Hereford, TX 


Linda Elliott 
Registrar 

West Texas A & M 
WT Box 192 
Canyon, TX 79016 


Margaret Kelo* 
Institutional Research 
West Texas A & M 
WT Box 192 
Canyon, TX 79016 


Lucy Walker 

Career & Technology Ed. 
Amarillo ISD 
7200 m-40 West 
Amarillo, TX 79106 


Gene Minor 
PEIMS Coordinator 
Amarillo ISD 
7200 IH-40 West 
Amarillo, TX 79106 


Dr. Christina Berry 
Institutional Research 
South Plains College 
1401 College Ave 
Levelland, TX 79336 


Sandy Stretcher 
Dir. of Special Populations 
Frank Philips College 
P.O. Box 5118 
Borger, TX 79008 


LeAnne Vogel 
Assistant Director 
Panhandle Tech Prep 
7200 IH-40 West 
Amarillo, TX 79106 


Dr. Stan Adetman, Dir.* 
Computer Services and 
Institutional Research 
Amarillo College 
P.O. Box 447 
Amarillo, TX 79178 


Tammy Smith 
Assistant Director 
Permian Basin QWFP 
PO Box 60660 
Midland, TX 79711 


Rebecca Alcazar 
Director 

South Plains QWFP 
P.O. Box 610 
Levelland, TX 79336 


Pam Zenick 
Planner 

Panhandle RPC 
P.O. Box 9257 
Amarillo, TX 79105 



Dr. Herlinda Coronado 
Dean of Instruaion 
South Plains College 
1401 College Ave 
Levelland, TX 79336 

^ indicates Steering Committee membership 



Marc Anderberg 
Program Direaor 
Automated Student Follow-Up 
Texas SOICC 

3520 Exec. Center Dr. #205 
Austin, TX 78731 
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Beaumont Site-Based Committee Membership List 



Dr. Carrie Nelson 
Deep East Tech Prep 
Angelina College 
P.O. Box 1767 
Lufkin, TX 75901 
(409) 633-5307 


Janis Hitchins, Dean 
Lamar U. - Port Arthur 
P.O. Box 310 
Port Arthur, TX 77641 
(409) 983-4921 (ext. 300) 


Dr. Ken Shipper, Exec. Dir. 
and M. Paul Roy 
P.O. Box 10043 
Beaumont, TX 77710 
(409) 880-8185 


Linda Brown-Turk, Exec. Dir.* 
Southeast Texas SDA 
P.O. Drawer 1387 
Nederland, TX 77627 
(409) 727-2384 


Ms. Brown-Turk*s staff: 
Marilyn Smith 
Delores Coleman 
Jackie Galloway 
Mary Hammon 


Patricia Duhon, Dir. 
Institutional Research 
Lamar U. - Beaumont 
P.O. Box 10601 
Beaumont, TX 77710 
(409) 800-8097 


Bobbie Burgess, Dean 
Student Services 
Lamar U. - Orange 
P.O. Box 11137 
Orange, TX 77630 
(409) 882-3341 


Debbie Holder, R.N. 
Silsbee High School 
415 W. Avenue N 
Silsbee, TX 77656 
(409) 385-5574 ext. 37 


Ray Brown, Director 
Southeast Texas Tech Prep 
Region V ESC 
2295 Delaware 
Beaumont, TX 77703 
(409) 835-5212 


Don Travis, Manager 
Southeast Texas QWFP 
Region V ESC 
2295 Delaware 
Beaumont, TX 77703 
(409) 835-5212 


Darylann Hansen* 

Dir., Computer Services 
Braumont ISD 
3395 Harrison 
Beaumont, TX 77706 
(409) 899-9972 





Marc Anderberg, Project Director, Texas SOICC 
3520 Executive Center Dr., Suite 205 
Austin, TX 78731 

(512) 502-3753 FAX: (512) 502-3763 



* Indicates membership on the Steering Committee 
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Dallas/Waco Site-Based Committee 



Dr. Ron Huffstutler (Committee Chair)* 

Vice President 

East Texas State University 

Commerce, TX 75429 

(903) 886-5919 FAX: 886-5918 




Dr. Sandra Neubert (Vice Chair)* 
Quality Assurance Director 
Waco ISD 
PO Box 27 
Waco, TX 76703 


Bobbie Crow 
The BOND Program 
1105 Woodland Acres #400 
Waco, TX 76712 


Dr. Lindle Grigsby 
Adult Vocational Ed. 
Brookhaven College (DCCCD) 
3939 Valley View Lane 
Farmer’s Branch, TX 75244 


Dr. iDarline Morris* 

Dir. of Institutional Reseaxch 
TSTC - Waco 
3801 Campus Dr. #32-9 
Waco, TX 76705 


Chris Lanham 
(replacing Dr. England) 
McLennan Community College 
1400 College Dr. 

Waco, TX 77060 


Richard Minter 
Dean 

Tyler Junior College 
PO Box 9020 
Tyler, TX 75711 


Robin Huskey 
Research Associate 
Kilgore College 
1100 Broadway 
Kilgore, TX 75662 


Anita 3teele 

Labor Market Analyst 

Interlink (North Central QWFP) 

PO Box 610246 

DWF Airport, TX 75921 


Karen Freiman 

Program Director 

Collin County PIC 

321 N. Central Expwy, #360 

McKinney, TX 75070 


Sheryl Kenney 
Research & Planning 
Tarrant Co. PIC 
2601 Scott Ave. #203 
Ft. Worth, TX 76103 


Sylvia Kelley 

Tech Prep Director 

Global Edge (Collin Co.) Tech Prep 

2200 W. University 

McKinney, TX 75070 


Jan Crews 
Tech Prep Director 
Texoma Tech Prep 
1525 W. California 
Gainesville, TX 76240 


Dr. Joe Mills, Director 
Institutional Effectiveness 
Trinity Valley Com. College 
500 Praireville St. 

Athens, TX 75751 


Don Perry* 

Dean 

Mountain View College (DCCCD) 
4849 Illinois Ave. 

Dallas, TX 75211 


Sharon Smith 
Institutional Research 
Dallas Co. Com. College Dist. 
701 Elm Street 
Dallas, TX 75202 


Linda Shoup 
Institutional Research 
Collin County Com. College 
2200 West University 
McKinney, TX 75070 


Dr. Mary Korfhage* 

Dir. Institutional Research 
U. of Texas - Dallas 
PO Box 830688 
Richardson, TX 75083 


Mary Ross* 

Planner 

West Central Texas COG 
PO Box 3195 
Abilene, TX 79604 


Robert D. Garvin 
Dir. Institutional Research 
Central Texas College 
PO Box 1800 
Killeen, TX 76540 



* Indicates membership on the Steering Committee 
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Houston Site*Based Committee Membership List 



Dn Mike Green* 

Institutional Research 
N. Harris/Montgomery CCD 
250 N. Beltway East 
Houston, TX 77060 
(713) 591-3521 


Rick Hernandez or Tina Parker 
Brazos Valley Tech Prep 
Vo Blinn College 
301 Post Office Rd. 

Bryan, TX 77803 
(409)823-4988 


Eileen Booher* 

Gulf Coast Tech Prep 
Vo N. HarrisAlontgomery CCD 
250 N. Beltway East 
Houston, TX 77060 
(713) 591-3531 


John Syers* 

Region IV ESC 
P. O. Box 863 
Houston, TX 77001 
(713) 744-6828 


Dr. Margaret Ford 
Asst. Vice Chancellor 
Houston Com. College System 
P.O.Box 7849 
Houston, TX 77270 
(713) 466-6654 


Dr. Ann Green 
Institutional Research 
Bee County College 
3800 Charco Rd. 
Beeville, TX 78102 
(512) 358-3130 


Dr. David Preston* 
Institutional Research 
Brazosport College 
500 College Drive 
Lake Jackson, TX 77566 
(409) 266-3000 


Roger Johnson or Beckie Colvin 
Golden Crescent Tech Prep 
Vo Victoria College 
2200 Red River Rd. 

Victoria, TX 77901 
(512) 572-6477 


Steve Johnson 
Goose Creek ISD 
Box 3 

Baytown, TX 77522 
(713) 420-4463 


Dr. Cynthia Dutschke, Dir. 
Institutional Research 
U of Houston - Clear Lake 
2700 Bay Area Blvd. 
Houston, TX 77059 
(713) 283-3006 


Franklin Higgins 
Career & Tech. Education 
Aldine ISD 

14910 Aldine-Whifield Rd. 
Houston, TX 77032 
(713) 985-6370 


Marc Anderberg, Project Dir. 
Automated Student Follow-Up 
Texas SOICC 

3520 Exec. Center Dr. #205 

Austin, TX 78731 

(512) 502-3754 FAX: 502-3763 


Dr. Richard Bailey* 
Institutional Research 
San Jacinto CC Dist. 

4624 Fairmont Pkwy #203 
Houston, TX 77502 
(713) 998-6176 


William Whitlow, Manager 
Technical & Support Srvc. 
Houston Works (Houston JTPA) 
1919 Smith, Suite 500 
Houston, TX 77002 
(713) 654-1919 





Indicates membersbip on the Steering Committee 
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APPENDIX VII (Continued) 

McAllen Site*Based Conunittee 

Leonard Theilin, Chairman* and Dr. Larry Kohler*, Paul Tate, Thomas Perez 
McAllen KD, 2000 N. 23rd St., McAllen, TX 78501 

Vickie C. Natale, Vice Chair* 

Institutional Research, Del Mar College, Baldwin and Ayers, 

Corpus Chisti, TX 78404 

Ms. Billie O’Dowdy* 

Workforce Development Corp., 5110 Wilkinson Dr. 

Corpus Chrisd, TX 78415 

Raul Garcia 

Cameron County PIC, 285 King’s Highway 
Brownsville, TX 78521 

Arturo McDonald, Jr. 

Lower Rio Grande Quality Workforce Planning, */, Short Course Center 
TSTC • Harlingen, Harlingen, TX 78550 

Arturo Meraz 

South Texas Quality Workforce Planning, Laredo Jr. College 
West End Washington Street, Laredo, TX 78040 

Dr. Victor Fuhro 

Dean of Occupational Education, Texas Southmost College 
80 Fort Brown Ave., Brownsville, TX 78520 

Pat Crouch (representing Allan Tipton) 

Institutional Research, Texas A&I University 
Campus Box 215, Kingsville, TX 78363 

Adrian Garcia, Jr. 

PEIMS Coordinator, Region I Ed Service Center 
1900 W. Schunior, Edinburg, 1X1 

Dr. Michael Metke 

Dean of Instruction, South Texas Community College 
3201 Pecan, McAllen, TX 78501 

Dr. Richard Sheppard 

Dept, of ^ucation, Texas A&M - Corpus Christi 
6300 Ocean Dr., Corpus Christi, TX 78417 



* Indicates membership on the Steering Committee 
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APPENDIX VIII (Continued) 
AGENCY LIAISON 



Bob Lahti, Helen Giraitis*, Ruben Garcia, Molly Boyd 
Community/Tcchnical College Division 
Higher Education Coordinating Board 

Kathy Benson, Information Services Division 
Higher Education Coordinating Board 

Lorraine Merrick, Dr. Mark Butler***, Judith Hetherly 
Career & Technology Education 
Texas Education Agency 

Karen Cornwell, Barbara Walters, Ted Brown 

PEIMS Division 

Texas Education Agency 

Jim Boyd, Robin Campbell*, Jim Gaston 
Work Force Development Division 
Texas Department of Commerce 

Leslie Geballe, Will Reece 
Intergovernmental Relations 
Texas Employment Commission 

Phoebe Knauer, Sandy Gebhart 
Information Release Division 
Texas Employment Commission 

Joanne Brown 

Travel and Conferences 

Texas Employment Commission 

Lynda Rife, Cynthia Mugerauer, Susan Hadley and Cindy Geisman 
Texas Council on Workforce & Economic Competitiveness 

Additions to the Steering Committee 

Dr. Mike Wolf, Vice President, El Paso Community College 
Dr. John Grablc*, President, Brazosport College 



* Indicates membership on the Steering Committee 
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APPENDIX VIII 
BUDGET CLOSE OUT 

SUBCONTRACTOR 

(Included Contract #110 and Amendment) 



PROJECT STAFF 


Salary/ 




Reconcile 




Fringes 


Total 


Arnount 


Jim Reed, Project Dir. 


$ 9,178 


$ 9,178 


$ 13,154 


Judy Reed, Associate Dir. 


4,145 


4,145 


7,690 


Support (progamming) 


2,157 


2,157 


3,194 


Support (coding/keying/mail) 


30,867 


30.867 


22.179 


Subtotal 




$46,347 


$ 46,937 


PROJECT STAFF TRAVEL 








Project Director/ Associate Dir. 




$ 5,150 


$ 3.911 


SUPPLffiS 




$10,270 


$ 3,451 


PRINTING/DUPLICATION 




$ 3,439 


$ 9,698 


COMPUTER/PERIPHERAL SUPPORT 


$10,744 


$ 12,876 


COMMUNICATIONS & POSTAGE 


$21,061 


$ 18,096 


GRAPHICS & SURVEY PRODUCTION 


$ 3,844 


$ 4,886 


DATA STORAGE MEDIA 




$ 1,530 


$ 2,530 


ADMINISTRATIVE COSTS (5%) 


$ 5.122 


$ 5.122 


TOTAL BUDGET 




$107,507 


$107,507 



Annual Reconciliation Notes: 

Because of billing procedures, some supplies were included in printing/duplication budget items. Additional 
personnel, computer, and data storage expenses were needed because of increased data and table maintenance 
responsibilities, data conversion, and other activities approved by subcontract manager. Shifts in funds from line 
items in budget approved by subcontract manager. 



O 
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APPENDIX VIII (Continued) 

GRANT RECIPIENT 

Operating Budget* 
1993-1994 Program Year 



ANTICIPATED SOURCES OF FUNDS 



Tri-agency Contributions 


Subtotals 


Totals 
$ 225,000 


Coordinating Board 


$ 75,000 




Texas Department of Commerce 


75,000 




Texas Education Agency 


75,000 




Program Income 




14,000 


Registration Fees: 






3 workshops @ $ 3,000 
1 statewide conference 


9.000 

5.000 




Processing Charges: 


N.A. 

(off-budget) 


cost recovery 



TOTAL INCOME $ 239,000 



* At time of print, closeout awaiting final bill from the Coordinating Board for data services. 
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PROPOSED EXPENDITURES 



Personnel and Related Expenses 


Breakouts 


Subtotals 


Salaries 




$ 54,150 


Projea Director (100%) 


$ 41,160 


Data Analyst (33%) 


12,990 




Fringe Benefits 




15,743 


Project Director 


12,348 


Data Analyst 


3,395 





Totals 
$ 69,893 



Travel and Related Expenses 16,700 

Orientation Meeting in Austin 
25 site committee members @ $ 200 5,000 

Steering Committee Meetings in Austin 

10 members * 3 meeting @ $200 6,000 

Director: 3 site visits * 5 sites @ $250 3,750 

Director: 3 regional workshops @ $250 750 

Director: other presentations 1,200 



Postage, Communications and Supplies 6^000 

Reproduction Costs 3^000 

Administrative Overhead 5,900 

SOICC overhead (1.7% of contract) 3,900 

14 of equipment for Vi time analyst 2,000 



Employer Survey Subcontract 
Original Contraa 
Amendment 


101,927 

5,580 


107,507 


Conference/ Workshop Expenses 

3 regional workshops @ $ 3,000 
1 statewide conference 


9.000 

5.000 


14,000 


THECB Processing, Programming and Hardware 




15,000 


TEC Record Matching Charges 




1,000 


Other Charges 


N.A 

(off-budget) 


cost recovery 



TOTAL EXPENDITURES 



* At time of print, closeout awaiting final bill from the Coordinating Board for data services, 
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ENDNOTES 



‘ Commission on the Skills of the National Workforce, America's Choice: high skills or low 
wa^es! (National Center on Education and the Economy; Rochester, NY: 1990). 

^ Ray Marshall and Marc Tucker, Thinking for a Living (Basic Books; New York City, New 
York: 1992). 

Hn computing rank order product moment correlation coefficients, economic data was avail- 
able only for the six pilot districts within Metropolitan Statistical Areas. Data from the two 
smallest districts, Silsbee and Hereford, were deleted from the analysis. 

^'Statistical significance is indicated in terms of the probability (p) of making an judgment in 
rejeaing the null hypothesis. Where p < .10, there is less than a 10% chance that the results 
could have been obtained in an absence of a relationship between the varialbes examined. The 
probability p < .10 could also be expressed as p > .90. 

®'Per capita income was available from the Bureau of Economic Analysis; median household 
income for each QWFP region from the Census Bureau. Wide dispersion around those 
medians rendered those figures incapable of capturing the tendencies of persons with 
comparable earnings to cluster in economically homogeneous communities or neighborhoods 
served by a school distria or campus. 

* Parental influence also effects recent graduates’ choice of pathways. One would hypothesize 
that graduates having one or both parents with some college education would be more likely 
to pursue higher education. However, parents’ educational attainment was not available in the 
seed records. If district data bases contain information about parents’ education, program ad- 
ministrators are advised to test that hypothesis. 

^ The Coefficient of Specialization indicates the degree to which a particular kind of business 
or industry is concentrated in a geographic area. 



/regional SIC employment / \ 

\ ' total regional employment/ 

/statewide SIC employment / \ 

\ • total statewide employment/ 



*'With a larger number of cases, it would be useful to run an analysis of variance or chi-square 
test on a crosstabulation of outcomes by gender and ethnicity using "economically disadvan- 
tagted" status as a control variable. 



’More detailed analysis would involve interposing economic status, gender, ethnicity, and 
student intent variables as stistical controls on the relationship between graduation type and 
post-graduation outcomes. 
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*° As defined in Perkins legislation. 



“ Minor discrepancies in spelling were ignored. Where last names matched, inversion of first 
name and middle initial or use of nicknames were ignored. Among females, if last names did 
not match, SSNs were assumed valid so long as the first name and middle initial matched ~ 
indicating a name change associated with a change in marital status. 

“ For want of a better term, "incidental" herein is used to denote students who take one or 
more career and technology education courses without regard to any coherent sequence, pro- 
grammatic design, or intent to improve occupational specific employment opportunities. (Also 
known as "taking courses cafeteria style.") Incidental vocational course-takers may enroll in 
a single course for avocational reasons, to fill a requirement, or to sharpen a skill not related 
to a particular desired career path. 

“In the CIP system, the last two digits of the eight digit course code may be dropped to 
indicate program enrollment. Successive deletion of training di^ts allows easy aggregation of 
data to department and division level categories. 

“ Baj and Trott of Northern Illinois University are currently funded by the DOL to compare 
traditional survey data to UI wage records in eleven states. While they have not yet released 
their findings, both researchers in private conversations with the System Director indicated 
that the findings made in Texas were entirely consistent with their observations in other states. 

“ Earnings of former JTPA participants overlap those for high school graduates, certificate 
holders and associate degree earners because there is a two year limit on eligiblity for Title IIA 
and Title III programs. Some Title IIA participants (economically disadvantaged adults) may 
enter the program without a high school diploma an^ in the two year limitation, may obtain 
a GED; others who enter with at least a high school diploma may advance in two years to 
earn either a certificate or an associates degree. Title III participants (displaced workers) tend 
to enter with higher levlels of prior education and are more likely to exit the program with 
some kind of postsecondary award. Becuase SDAs purchase off-the-shelf training from com- 
munity and technical colleges for JTPA participants, these two columns do not represent 
unduplicated figures. 

“Memorandum from Barbara Ann Farmer, Administrator for Regional Management, DOL 
Training nad Employment Information Notice No. 38-93 (Washington, DC: March 3, 1994), 
page 2. 
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GLOSSARY 



CIP 



Coefficient of 
Specialization 



Coordinating 

Board 



DOL 



DOT 



ESL 

ESC 



FICE 



Dependent 

Variables 

GPRA 



Classification of Instructional Programs. A standardized coding system 
developed by the US Department of Education. The code assigned to a 
class can be truncated or collapsed in ascending order to represent 
program, department and division. 

A statistic indicating the degree to which industrial employment is con- 
centrated in a geographic area relative to a larger, self-sustaining or 
independent geographic area such as a state or the nation. 

Texas Higher Education Coordinating Board. The state’s central agency 
for higher ^ostsecondary) education and is responsible for the 
administration of a proportionate share of federal Perkins dollars. 

Department of Labor, the federal agency responsible for administering 
employment and training programs (including JTPA and education coor- 
dination), collecting employment data, and making employment 
forecasts. 

Dictionary of Occupational Titles -, a coding system for classifying 
occupational titles according to the type of work performed. Although 
it covers more titles than the OES system and provides more detailed 
analyses of the work performed, the DOT is not used to code current 
employment levels or in forecasting occupational employment demands. 

English as a Second Language, a special populations category under the 
Perkins Act. 

Education Service Center, any one of several offices (designated by 
region number) that provides technical assistance to local education 
agencies. 

Federal Identification Code for Education: a standardized code for identi- 
fying education and training institutions certified to receive federal funds 
or to provide services to participants receiving federal assistance. 

An event or phenomenon that needs to be explained. In the case of 
automated follow-up, outcomes constitute the dependent variables. 

Government Performance and Results Act of 1993 (Public Law 103-62). 
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Independent 

Variables 


Background, antecedent or intervention/treatment information believed 
or hypothesized to have some capacity to explain variance in the 
dependent variable(s). In the case of automated follow-up, the 
information contained in the seed records and among the data elements 
extracted at the option of program administrators from an in-house 
management information system. 


ISD 


Independent School District, local education agencies across the state. (In 
some locations may be known as CSDs, "Consolidated School 
Districts.") 


JSEC 


Job Service Employer Committees; conunittees of employers formed in 
each TEC region to render advice about job service and employment 
research activities. 


JTPA 


Job Training Partnership Act (PL 97-300 amended in 1992 by PL 94-404) 
administered in Texas by the Texas Department of Commerce. 


LEP 


Limited English Proficiency: a special populations category under 
Perkins Aa. 


Match 


A match occurs when a unique identifier is found in two or more linked 
data bases. Also known as a "hit." 


Master Plan 


State Master Plan for Career and Technology Education; a plan 
developed by the tri-agencies to improve the integration of workforce 
development programs; pre-dates the Strategic Plan. 


NHMCC 


North Harri«/Montgomery County Community College, the first Texas 
institution of higher education to use record linkage techniques to 
identify labor market outcomes of its former students. 


NOICC 


National Occupational Information Coordinating Committee, largely 
responsible for the development of crosswalks between coding systems. 


OES 


Occupational Employment Statistics, a coding system used by the US 
Department of Labor and state employment service agencies in coding 
occupations and collecting information on staffing patterns and future 
occupational employment needs of employers. The JTPA system in 
Texas uses the OES coding system in its MIS. 


P < 


In statistics, the probability of making an erroneous judgment in 
accepting or rejecting the null hypothesis. 
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Perkins 

Act 


Carl D. Perkins Vocational and Technical Education Act (PL 98-524 as 
amended by PL 100-392). In Texas, federal Perkins dollars are 
administered by the Texas Education Agency (secondary) and the Texas 
Higher Education Coordinating Board (postsecondary). 


QWFP 


Quality Work Force Planning; a network of regional planning 
committees formed with seed money from the tri-agencies to help 
integrate the delivery of career and technology education and training. 


Right to Know 


Student Right to Know Act and Campus Security Act of 1990, Public 
Law 101-542. 


SACS 


Southern Association of Colleges and Schools, a multi-state institution 
accrediting body. 


SB 642 


Senate Bill 642: The Texas Workforce and Economic Committee Act of 
1993. This bill created the Texas Council on Workforce and Economic 
Competitiveness with a mandate to develop and integrated, compre- 
hensive statewide follow-up system to gather information for the 
purpose of planning and evaluating publicly funded workforce 
development programs across the state. 


Seed Record 


Background information on a former student or participant including 
Social Security number, demographic information, program participation 
and/or completion status, certain special populations status. Such items 
constitute the principle independent variable used in disaggregating and 
explaining variance in outcomes. 


SIC 


Standard Industrial Classification used by the US Department of Labor 
and most state and local work force development entities to group firms 
into a hierarchical system based on similarity of products produced or 
services rendered. 


SMA 


Statistical Metropolitan Areas. 


SOICC 


The Texas State Occupational Information Coordinating Committee, 
follow-up grant recipient for Program Years 1992-93, 1993-94, and 1994- 
95; charged with responsibilities under Senate Bill 642 and the TCWEC’s 
state strategic plan for developing and operating a comprehensive, 
statewide integrated follow-up system. 
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special Pop* 
ulations 

Strategic Plan 

TCWEC 

TDoC 

TEA 

TEC 

THECB Student 
Record 

Tri-Agencies 
UI Wage Record 



Subgroups identified for targeted services for programs using federal 
Perkms dollars (also known as -Special Pops"). In public education, these 
include student with limited English proficiency (LEP), bi-lingual and 
imimgrant students smd those for whom English is a Second Language 
(ESL), economically disadvantaged students, academically disadvantaged 

teenaged mothers, incarcerated, gender equity, and 
those classified as Special Education students. 



State Strategic Plan for Workforce and Economic Competitiveness under 
development pursuant to mandates in Senate Bill 642 by a blue ribbon 
task force in collaboration with the TCWEC, TCWEC staff, and work- 
force development partner agencies. 

Texas Council on Workforce and Economic Competitiveness, created 
under Senate Bill 642 to facilitate coordinated planning, budgeting, 
imp mentation and evaluation of the federally funded workforce 
development programs ih Texas. 



Texas Department of Commerce. In Texas, with the exception of an 8% 
set Mide for coordination, this agency is responsible for the adminis- 
tration of federal JTPA dollars. 



Texas Eduction Agency. The state’s central education agency for public 
education (K-12) and is responsible for the administration of a propor- 
tionate share of federal Perkins vocational dollars flowing to the state. 

Texas Employment Commission; collects quarterly wage reports on 
workers covered under the Texas Unemployment Compensation Act; 
rpponsible for release of information on employment outcomes; also 
fiscal agent for the Texas SOICC. 

All public institutions of higher education report enrollment data by 
bocial Security number each semester/term for all academic and technical 
program students This data base is used to identify which students 
tracked by the follow-up system are pursuing additional education. 

Prior to passage of Senate Bill 642, the cooperative workforce 
development efforts of TEA, the Coordinating Board and TDoC. 

Unemployment Insurance quarterly reports used by the TEC to verify 
entitlement to benefit levels paid to claimants under the state’s 
unemployment compensation act; covers approximately 97% of all 
workers in Texas. 
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